Archive

Author Archive

The domain proof hash token (TXT record) can be deleted after Office 365 hybrid configuration is complete

I recently came across an interesting scenario at one of the most prestigious UK public sector organizations (without saying any names). I found that during Office 365 Hybrid Configuration, someone had prepended the domain validation hash token to the existing SPF TXT record instead of creating a new TXT record.

I did find that they had good reason to do so as the DNS servers used for public DNS were so antiquated they did not support multiple TXT records for the same domain. Adding the domain validation token text to the SPF TXT allowed the hybrid configuration to be set up but broke the SPF record.

To correct the SPF record as I advised, the ‘Change Advisory Board’ required authoritative confirmation direct from the horse’s mouth, i.e., Microsoft, that removing the domain validation token from the SPF TXT record would not have any desirable effect on the hybrid configuration.

Although I had full knowledge that the domain validation token had no purpose after the federation trust had been set up, I obtained the required official authoritative confirmation from an ex-colleague at Microsoft. Yes, I did work at Microsoft quite recently, until April 2014).

Therefore, having had this official confirmation, I spell it out for those of you who are still unsure. The domain validation TXT records can be safely removed after the federation trust has been set up. This is not known to have any undesirable effects on the hybrid configuration or the federation trust.

Go on, delete that TXT record. I have deleted mine.

Deleted forever!!

Deleted forever!!

Disable LANMAN using Group Policy

April 19, 2012 3 comments

—DISCLAIMER—
This is a rapid publishing post and may not include all necessary links and references and is not likely to meet the quality standard you and I expect of my blog posts.

SCENARIO:
A client company had a network and systems vulnerability testing done and were asked to disable storage of LANMAN hashes and LANMAN authentication to pass the audit.

I expect the audience of this article to have a basic understanding of authentication in Windows based networks and familiarity with the words LANMAN, NTLM and Kerberos is expected. There is plenty of information available online covering why LANMAN is bad so I’ll not duplicate that and get straight to the process to follow to disable the use of LM.

SOLUTION:
There are two parts to the solution.

Disabling the storage of LM Hashes

  1. Create a group policy object “NoLmHash” or call it whatever your preference or naming convention dictates.
  2. Navigate to Computer Configuration\Policies\Windows Settings\Local Policies\Security Options.
  3. Enable the setting “Network Security: Do not store LAN Manager has value on next password change.”

The above will not clean up any previously stored LM hash values and only means Windows will not compute and store new LM hash values for new passwords. You may also want to note that this setting is already included in the Default Domain Policy in a new Windows Server 2008 R2 domain. Also, Windows Vista and Windows 7 (also Windows Server 2008 and Windows Server 2008 R2) have LM hashes disabled by default and can be confirmed by navigating to registry value “NoLmHash” under HKLM\System\CurrentControlSet\Lsa.

Disabling LM Authentication

  1. Create a group policy object “NoLmAuthClient” as below and assign it to all computers except DCs.
  2. Navigate to Computer Configuration\Policies\Windows Settings\Local Policies\Security Options.
  3. Enable the setting “Network Security: LAN Manager Authentication Level” and set it to “Send NTLM response only”. Microsoft explanation for this setting, self explanatory as it is, is “Clients use NTLM authentication only and use NLTMv2 session security if the server supports it; domain controllers accept LM, NTLM and NTLMv2 authentication.”

To add further clarity, the above will prevent all clients from using LM, but DCs will continue to accept LM.

  1. To stop the DCs from accepting LM, create a group policy object “NoLmAuthDC” as below and assign it to all domain controllers.
  2. Navigate to Computer Configuration\Policies\Windows Settings\Local Policies\Security Options.
  3. Enable the setting “Network Security: LAN Manager Authentication Level” and set it to “Send NTLMv2 response only\refuse LM”. Microsoft explanation for this setting is “Clients use NTLMv2 authentication only and use NTLMv2 session security if the server supports it; domain controllers refuse LM (accept only NTLM and NTLMv2 authentication).”

The above will prevent the DCs from accepting LM. There’s more to play with in the same GPO area and you can configure LM auditing. I’ll leave it to your brilliant minds and capable hands; it shouldn’t be very hard from here on.

You can also go a step further and disable NTLM as well and allow just NTLMv2. It’s different values of the same setting “Network Security: LAN Manager Authentication Level”. You’ll get on just fine but send any questions in comments if needed.

RISKS/CAVEATS

These settings may cause and an issue where you have Windows NT4.0 or older clients on your network. But since very old versions of Windows are not held on to with much love for years, I wouldn’t worry too much about it, not for networks I manage.

OTHER USEFUL INFORMATION

  • Windows 95 and Windows 98 do not support NTLM.
  • Windows NT 4.0 earlier than SP4 does not support NTLMv2 and all Windows versions since Windows NT 4.0 SP4 support NTLMv2.
  • LM is disabled by default in Windows Vista and Windows 7.
  • LM uses LM hash which is the least secure way of storing a password in Windows.
  • NTLM credentials consist of a domain name, a username and a one-way hash of the user’s password.
  • NTLM does not support AES or SHA-256. It uses CRC for integrity and RC24 for encryption.
  • NTLM uses an encrypted challenge/response protocol and does not send the password over the wire.
  • All Windows versions since Windows 2000 support Kerberos.
  • Kerberos is the preferred authentication protocol for Windows 2000 and subsequent Active Directory domains.
  • Kerberos is a domain only protocol, NTLM is used if authenticating to a system that is not on the domain.
  • Kerberos requires a server name, so NTLM is used if a client is authenticating to a server using an IP address.
  • Kerberos makes use of many different ports so passing it through firewalls can be fun if not previously familiar with it.
  • NTLM, NTLMv2 and Kerberos all use the NT hash, also known as the Unicode hash.

BES Services to restart automatically after Exchange reboot

February 16, 2011 3 comments

If you have been managing Blackberry Enterprise Servers for some time, you’re probably familiar with the well known problem of Blackberries not being able to send/receive emails after an Exchange server reboot unless the BES server is also rebooted or the Blackberry services are restarted.

What happens in the background is that BES loses MAPI connections to the Exchange server and is not able to re-establish this once the Exchange services come back online. Exchange service restarts can happen for any reason, Windows update, someone manually restarting the server and forgetting there’s a Blackberry server to reboot as well. (Does this remind you of someone?)

If your BES uptime monitoring is like most people, you probably have telephone alerts set up. It’s a sophisticated kind of monitoring where an annoyed user rings up saying, “My Blackberry isn’t working. I thought this was being monitored?” You come up with something nice to say, probably apologize for this happening too often (what else could you say, customer is king!) and tell them the Blackberry should start working in no time. Then you restart the Blackberry services, and everything is hunky dory again. Well, until next time.

But can you really do this every other day? Especially if you’re a managed service provider managing dozens or hundreds of Blackberry servers?

SOLUTION

I was asked to attack this problem and I took two approaches, proposals for both are under technical review at the moment with our NOC team.

1. Add intelligence to the Exchange Server to automatically restart Blackberry services 5 minutes after it starts up.

This solution involves setting up a startup script on Exchange Server. This script will use psexec to remotely execute a script on the Blackberry server which will restart the Blackberry services.

However, before calling the remote restartBes.bat script using psexec, it will call a local VBScript Wait.vbs which will do what? It’ll make execution wait for 5 minutes so Blackberry services will be restarted after a delay of 5 minutes. This allows the Exchange Server to fully start up, see the network and start its services first. You can modify the delay as per your environment. The code below is the code you use for startup script on the Exchange Server.

rem *** Calling Wait.vbs to add a delay of 5 minutes ***
C:\Scripts\Wait.vbs

rem *** Calling psexec to remotely execute restartBes.bat on Blackberry Server ***
C:\PSTools\psexec \\BesServer -s C:\Scripts\restartBes.bat
rem *** Execution passed to restartBes.bat. Quitting. ***

The code below is the code you will use for Wait.vbs which will need to be placed in C:\Scripts on the Exchange Server. 300000 is the wait time in milliseconds which you can change.

WScript.Sleep 300000

The code below is the code you will use for RestartBes.bat which will need to be placed in C:\Scripts on the Blackberry Server. The correct order to stop and start Blackberry services has been taken from Blackberry KB13718.

rem *** Stopping Services ***
net stop “BlackBerry Controller” /yes
net stop “BlackBerry Dispatcher” /yes
net stop “BlackBerry Router” /yes
net stop BAS-NCC /yes
net stop BAS-AS /yes
net stop “BlackBerry Server Alert” /yes
net stop BBAttachServer /yes
net stop “BlackBerry MailStore Service” /yes
net stop “BlackBerry MDS Connection Service” /yes
net stop BBMonitoringConsole /yes
net stop BBMonitoringService_APP /yes
net stop BBMonitoringService_DCS /yes
net stop “BlackBerry Policy Service” /yes
net stop BlackBerry SyncServer /yes
net stop BBMonitoringService_ENG /yes

rem *** Starting Services ***
net start “BlackBerry Router” /yes
net start “BlackBerry Dispatcher” /yes
net start “BlackBerry Controller” /yes
net start BAS-AS /yes
net start BAS-NCC /yes
net start “BlackBerry Server Alert” /yes
net start BBAttachServer /yes
net start “BlackBerry MailStore Service” /yes
net start “BlackBerry MDS Connection Service” /yes
net start BBMonitoringConsole /yes
net start BBMonitoringService_APP /yes
net start BBMonitoringService_DCS /yes
net start “BlackBerry Policy Service” /yes
net start BlackBerry SyncServer /yes
net start BBMonitoringService_ENG /yes

You may or may not have the Blackberry Monitoring Service installed so 4 of these services may not exist on your system but the script will work regardless.

2. Use an event log monitoring tool to look for events that are logged when this problem happens and restart Blackberry service when the relevant events are logged.

Blackberry Enterprise Server 5 generates application log events 20709 “Failed to reach user’s mailbox” for every user. The idea is for the monitoring tool to restart Blackberry services if this event is seen more than X number of times in Y number of minutes, say 5.

Additionally you can configure your monitoring tool to send an email when this happens or if it has to restart the services more than 2 times in a 30 minute period which shows there may be a bigger problem needing human intervention.

It’s easy to do this with any good Remote Monitoring & Management (RMM) tool. We have used Kaseya in the past and use Labtech now.

ADDITIONAL MONITORING POSSIBILITIES

As some of you may be aware, the Blackberry Monitoring Service uses SNMP. You can set up an SNMP manager of your choice and set up SNMP monitoring for your Blackberry Enterprise Server. Personally, I like PRTG Network Monitor which is used by enterprises all around the world, although there are many other options out there. I may make an additional blog post later about setting up PRTG or monitoring Blackberry Enterprise Servers with PRTG. Subscribe to my RSS Feed if you’d like an update.

APPLIES TO

This blog post was written for Blackberry Enterprise Server 5. Most sysadmins should be able to make this work for other versions of BES.

P.S. I work with Blackberries but I think they are ugly and so last century. BB Torch? Too much of an “I want to be an iPhone” personality there!

Scripted ftp download of zipped SQL DB, restore into SQL Server

February 14, 2011 4 comments

Recently I was given a requirement by a client to set up a scheduled ftp download of a zipped Microsoft SQL Server database backup file from a 3rd party’s ftp server and restore it into an SQL Server instance on one of the client’s servers. The process was to be run every morning.

I approached the problem using the below steps

  1. Connect to 3rd party’s server using ftp and download the .zip file containing the backup of the Microsoft SQL Server database.
  2. Unzip the file to extract the .bak file containing Microsoft SQL Server database backup.
  3. Detach existing Microsoft SQL Server database and delete it’s .mdf and .ldf files.
  4. Restore .bak file as new database.

SOLUTION

Here’s how you’ll achieve the same. This procedure uses some SQL scripts and one or more utilities not present in a default Windows installation. I’ll provide details for these as well in the dependencies section below.

1. Create a folder called ‘Restore’. This folder will store the relevant scripts and this is where the downloading/unzipping will happen.

2. In this folder, create a file ‘DailyRestore.bat’ with below commands which you’ll modify to match your server\instance name and file and folder locations. This is the main batch file that does all the process.

@echo off
echo Deleting previously downloaded .zip and .bak files
del M:\Restore\*.zip
del M:\Restore\*.bak

echo Connecting to ftp using commands from ftp.txt and downloading .zip file, then renaming it to DB.zip
ftp -s:M:\Restore\ftp.txt ftp.3rdparty.net
rename M:\Restore\*.zip DB.zip

echo Unzipping compressed backup
unzip M:\Restore\DB.zip
rename M:\Restore\*.bak DB.bak

echo Detaching previous database
sqlcmd -S SERVERNAME\INSTANCENAME -i detachDB.sql

echo Deleting previous database files
del “C:\Program Files\Microsoft SQL Server\MSSQL.3\MSSQL\Data\DB_Live.mdf”
del “C:\Program Files\Microsoft SQL Server\MSSQL.3\MSSQL\Data\DB_Live_log.ldf”

echo Restoring database from new backup
sqlcmd -S SERVERNAME\INSTANCENAME -i restoreDB.sql

3. Schedule the script to run at the time of your choosing.

DEPENDENCIES

All dependencies need to be saved in the ‘Restore’ folder you created earlier.

1. ftp.txt – This file includes ftp commands to connect to an ftp server and get a file. Use text below – modify as you need. Each line is a command that the ftp utility will send to server. Yes, this includes username/password.

ftpuser
ftp password
cd database
mget *.ZIP
y
bye

2. unzip.exe – download from here

3. detachDB.sql – you’ll need to modify this to suit your own SQL installation. You can modify this or create your own detachDB script by manually detaching DB in SQL Server Management Studio and doing ‘Script Action to file’ instead of actually detaching the DB.

USE [master]
GO
EXEC master.dbo.sp_detach_db @dbname = N’DB_Live’, @keepfulltextindexfile=N’true’
GO

4. restoreDB.sql – again you’ll need to modify this or create your own.

RESTORE DATABASE [DB_Live] FROM  DISK = N’M:\Restore\DB.BAK’ WITH  FILE = 1,  MOVE N’SQLMASTER_Data’ TO N’C:\Program Files\Microsoft SQL Server\MSSQL.3\MSSQL\Data\DB_Live.mdf’,  MOVE N’SQLMASTER_Log’ TO N’C:\Program Files\Microsoft SQL Server\MSSQL.3\MSSQL\Data\DB_Live_log.ldf’,  NOUNLOAD,  STATS = 10
GO

CONSTRAINTS

  1. The ftp commands assume the presence of only one .zip file to download in the ‘database’ folder. In my client’s case, the 3rd party overwrites the file every day.
  2. This was written for and tested with Microsoft SQL Server 2005 Express on Windows Server 2003. No warranties are made to work with this version of Windows and SQL Server or any other.