Tech Talk by Mohsin Abbas

If you have been managing Blackberry Enterprise Servers for some time, you’re probably familiar with the well known problem of Blackberries not being able to send/receive emails after an Exchange server reboot unless the BES server is also rebooted or the Blackberry services are restarted.

What happens in the background is that BES loses MAPI connections to the Exchange server and is not able to re-establish this once the Exchange services come back online. Exchange service restarts can happen for any reason, Windows update, someone manually restarting the server and forgetting there’s a Blackberry server to reboot as well. (Does this remind you of someone?)

If your BES uptime monitoring is like most people, you probably have telephone alerts set up. It’s a sophisticated kind of monitoring where an annoyed user rings up saying, “My Blackberry isn’t working. I thought this was being monitored?” You come up with something nice to say, probably apologize for this happening too often (what else could you say, customer is king!) and tell them the Blackberry should start working in no time. Then you restart the Blackberry services, and everything is hunky dory again. Well, until next time.

But can you really do this every other day? Especially if you’re a managed service provider managing dozens or hundreds of Blackberry servers?

SOLUTION

I was asked to attack this problem and I took two approaches, proposals for both are under technical review at the moment with our NOC team.

1. Add intelligence to the Exchange Server to automatically restart Blackberry services 5 minutes after it starts up.

This solution involves setting up a startup script on Exchange Server. This script will use psexec to remotely execute a script on the Blackberry server which will restart the Blackberry services.

However, before calling the remote restartBes.bat script using psexec, it will call a local VBScript Wait.vbs which will do what? It’ll make execution wait for 5 minutes so Blackberry services will be restarted after a delay of 5 minutes. This allows the Exchange Server to fully start up, see the network and start its services first. You can modify the delay as per your environment. The code below is the code you use for startup script on the Exchange Server.

rem *** Calling Wait.vbs to add a delay of 5 minutes ***
C:\Scripts\Wait.vbs

rem *** Calling psexec to remotely execute restartBes.bat on Blackberry Server ***
C:\PSTools\psexec \\BesServer -s C:\Scripts\restartBes.bat
rem *** Execution passed to restartBes.bat. Quitting. ***

The code below is the code you will use for Wait.vbs which will need to be placed in C:\Scripts on the Exchange Server. 300000 is the wait time in milliseconds which you can change.

WScript.Sleep 300000

The code below is the code you will use for RestartBes.bat which will need to be placed in C:\Scripts on the Blackberry Server. The correct order to stop and start Blackberry services has been taken from Blackberry KB13718.

rem *** Stopping Services ***
net stop “BlackBerry Controller” /yes
net stop “BlackBerry Dispatcher” /yes
net stop “BlackBerry Router” /yes
net stop BAS-NCC /yes
net stop BAS-AS /yes
net stop “BlackBerry Server Alert” /yes
net stop BBAttachServer /yes
net stop “BlackBerry MailStore Service” /yes
net stop “BlackBerry MDS Connection Service” /yes
net stop BBMonitoringConsole /yes
net stop BBMonitoringService_APP /yes
net stop BBMonitoringService_DCS /yes
net stop “BlackBerry Policy Service” /yes
net stop BlackBerry SyncServer /yes
net stop BBMonitoringService_ENG /yes

rem *** Starting Services ***
net start “BlackBerry Router” /yes
net start “BlackBerry Dispatcher” /yes
net start “BlackBerry Controller” /yes
net start BAS-AS /yes
net start BAS-NCC /yes
net start “BlackBerry Server Alert” /yes
net start BBAttachServer /yes
net start “BlackBerry MailStore Service” /yes
net start “BlackBerry MDS Connection Service” /yes
net start BBMonitoringConsole /yes
net start BBMonitoringService_APP /yes
net start BBMonitoringService_DCS /yes
net start “BlackBerry Policy Service” /yes
net start BlackBerry SyncServer /yes
net start BBMonitoringService_ENG /yes

You may or may not have the Blackberry Monitoring Service installed so 4 of these services may not exist on your system but the script will work regardless.

2. Use an event log monitoring tool to look for events that are logged when this problem happens and restart Blackberry service when the relevant events are logged.

Blackberry Enterprise Server 5 generates application log events 20709 “Failed to reach user’s mailbox” for every user. The idea is for the monitoring tool to restart Blackberry services if this event is seen more than X number of times in Y number of minutes, say 5.

Additionally you can configure your monitoring tool to send an email when this happens or if it has to restart the services more than 2 times in a 30 minute period which shows there may be a bigger problem needing human intervention.

It’s easy to do this with any good Remote Monitoring & Management (RMM) tool. We have used Kaseya in the past and use Labtech now.

ADDITIONAL MONITORING POSSIBILITIES

As some of you may be aware, the Blackberry Monitoring Service uses SNMP. You can set up an SNMP manager of your choice and set up SNMP monitoring for your Blackberry Enterprise Server. Personally, I like PRTG Network Monitor which is used by enterprises all around the world, although there are many other options out there. I may make an additional blog post later about setting up PRTG or monitoring Blackberry Enterprise Servers with PRTG. Subscribe to my RSS Feed if you’d like an update.

APPLIES TO

This blog post was written for Blackberry Enterprise Server 5. Most sysadmins should be able to make this work for other versions of BES.

P.S. I work with Blackberries but I think they are ugly and so last century. BB Torch? Too much of an “I want to be an iPhone” personality there!