In this post we share how Hillingdon Council is using the Common Technology Services (CTS) secure email guidance to change the way we secure email, and some of the lessons we learned along the way.
A bit about us
Hillingdon is the second largest London borough. Over the last few years the Council has done a major programme to refurbish and revitalise facilities for residents, so it was natural that we looked to improve our own internal infrastructure as well.
The Government Connect Secure Extranet (GCSx) is the secure private email service used by local authorities to share information with other public sector organisations. In summer 2015 we were informed that there would be changes to the GCSx and towards the end of 2015 we decided to be an early adopter of the secure email guidance from Common Technology Services (CTS).
We had several reasons for the change. We have 3,000 users with approximately 600 of them on the GCSx email. This meant that they had to use two email clients, and deal with complex rules about which email they could use for different types of work. Also, the ageing email system did not ‘fit’ with the rest of our business which we had already moved to Google Apps in June 2012. If we could get the security right then we would benefit from moving the GCSx domain to Google Apps and removing the old infrastructure. This would simplify email for all our users.
Getting from A to B
The key challenge for us was making sure that all our email gets delivered. Historically, we’ve used suppliers offering a ‘send as you’ service - this is where they would use their own email infastruture, but our hillingdon.gov.uk domain name as part of the email address - which was useful at the time. But with the reputational Domain Name System (DNS) controls required as part of the secure email guidance, we anticipated an impact on these ‘legitimate spoofers’ and knew there was a risk that some email could be marked as spam or simply deleted.
Taking our time and paying attention to detail was the only way to approach this. With a slow increase of the Domain-based Message Authentication, Reporting & Conformance (DMARC) record and regular reviews of the DMARC reports, we kept the risks low.
Doing it this way meant we also had time to review how we build our user’s devices. For example we rebuilt our devices with Windows 7, then tested them to ensure they met the necessary standards published by CESG. We also used this time to consider and construct our end user communications for when we’d reached the point of “good to go” to ensure everyone knew what we were doing and why we were doing it.
DMARC and the Spoofers
To get DMARC running smoothly there are some things to set up:
- Sender Policy Framework (SPF) lets you list valid sources for your domain’s email and publish it in a DNS record. We already had this set-up but it never hurts to check it is up to date.
- Domain Keys Identified Mail (DKIM) is used to digitally sign email and protect it from tampering in transit. Our Google Apps email service generates this for us, but we had to make sure that our email proxy and filtering supported DKIM signing as well. This meant upgrading our Clearswift service to the latest version to make sure it worked.
Finally we set up DMARC, which had to be done carefully. If we’d been too strict too early then email to residents might not have reached them. If email to residents didn't carry the correct DKIM signature they could be filtered as spam.
You can implement a DMARC ‘report only’ record at any time without affecting email flow, so with with SPF checked and DKIM ready, we set our DMARC record to p=none:
v=DMARC1;p=none;adkim=r;aspf=r;fo=1;rua=mailto:dmarc-dmarc@hillingdon.gov.uk,mailto:dmarc-rua@dmarc.service.gov.uk;ruf=mailto:dmarc-dmarc@hillingdon.gov.uk,mailto:dmarc-ruf@dmarc.service.gov.uk
After a few days we started to get reports about email using our domain. These reports covered both messages from declared systems and those that didn’t pass the SPF and DKIM checks. We left it this way for around 3 weeks to help us identify senders who looked legitimate, but didn’t come from our email service.
It was good that we started with ‘report only’ because our Internet Service Provider (ISP) active monitoring service was picked up as spoof email as it didn’t come from our server. We also picked up an email marketing service (Mailchimp) which was being used legitimately but hadn’t been included on our SPF record. We added both of these to the record so they could continue to send email on our behalf. At this point we could go to the business with firm information and the knowledge we could fix these business processes. This also presented a good opportunity to tidy up business systems sending email using our domain name (hillingdon.gov.uk) that did not use our email servers.
Once confident we had identified all the legitimate sending sources, we set our DMARC record to p=quarantine with pct=10:
v=DMARC1;p=quarantine;pct=10;adkim=r;aspf=r;fo=1;rua=mailto:dmarc-dmarc@hillingdon.gov.uk,mailto:dmarc-rua@dmarc.service.gov.uk;ruf=mailto:dmarc-dmarc@hillingdon.gov.uk,mailto:dmarc-ruf@dmarc.service.gov.uk
We left it like that for a couple of weeks before moving to pct=25, and started using the forensic reporting to dig into specific failure reports. Again, after another couple of weeks we set pct=50.
Interestingly, at each step we found new notifications from third parties we were not aware of, so we’re in the process of working with the various business areas and third parties to decide the best way to resolve these issues.
As of late Sept 2016 we are still rolling out DMARC by slowly raising the percentage of messages to which it applies.
Top tips
A practical point is that you need both the digest and forensic DMARC reports. The digest report is a summary, and while useful it’s helpfulness in tracing spoofers is limited in terms of data it contains. Some systems don’t publish forensic reports due to privacy issues, so you may have to investigate a way of getting the detailed reports.
Remember that you may be trying to trace untouched messaging solutions that have been in place for years. You may be accessing them through a customised web page that gives no clue to its links or how it operates.
They may be part of business processes that have worked well for years and become a forgotten part of business as usual. In this situation, the systems are effectively generating spoofed messages. They are from your domain but end up in spam because they aren’t included in your SPF record or DKIM signed. To reduce this risk spread the word to as much of the business as possible. Tell them what you are doing and why, and explain that these emails are now a problem.
Look out for syntax issues in DMARC DNS records as the text record will limit your entry length. The single line text limit in DNS is 255 characters. We had a few issues with syntax using multiple email addresses. We solved it by putting in digest (rua) and forensic (ruf) tags once and used commas to put in multiple email addresses.
Finally, it’s impossible to make too many notes on what you did, such as test outputs on end user devices, to help you keep track of what you’ve done.
Support
It was good to have support from Common Technology Services (CTS) during this process. They were happy to provide help via email or do a site visit if we wanted to check anything. It was useful to explore the next stages with them and work out the best ways of approaching some of the technical aspects.
Get in touch
If you are using the secure email guidance, or planning to soon, we’d love to hear from you. You can contact us.
Bruce Thomson is the IT Security Manager for the London Borough of Hillingdon and Julian Hamlin is the IT Security Appliance Manager.