Finding Yourself On The Spamhaus Blocklist

We recently had our company’s IP added to the Spamhaus email blocklist, resulting in fairly disruptive mail delivery failure. So I thought I’d detail some of the processes we went through in identifying the cause of this, some of the mistakes we made to let this happen in the first place, as well as some of my personal opinions on Spamhaus.

What Is Spamhaus?

Spamhaus is a popular, free, publicly accessible anti-spam list, allowing anybody to query its database of “known” spammer IPs. It’s a list used by many appliances, both as a definitive blocklist and as an influencing factor in determining the legitimacy of inbound email.

Identifying The Issue

The first sign of a problem was, of course, an increase in helpdesk calls and tickets for undeliverable mails (“The emails is broken!”). The fact that not all external mails were effected caused some initial confusion, but a quick look at the NDR made it plainly obvious what the problem was:

Remote Server returned '<remote-domain.mail.protection.outlook.com #5.7.1 SMTP; 550 5.7.1 Service unavailable, Client host [123.123.123.123] blocked using Spamhaus. To request removal from this list see https://www.spamhaus.org/query/ip/234.234.234.234 AS(1440)>'

I was familiar with Spamhaus from earlier this year when trialing DNSBL lists on a new FortiMail appliance. I seem to recall deciding against using Spamhaus, as it was very sensitive, blocking mails that no other DNSBL lists or FortiGuard were detecting as malicious.

The Blocklists

On visiting the spamhaus.org site, you’re met with a frankly confusing array of lists; SBL, CSS, XBL, PBL, DBL, ROKSO…

The most important lists if you find your address blocked are the SBL, PBL and XBL, which are defined as follows:

1. SBL (Spamhaus Block List): Verified sources of spam emails.
2. PBL (Policy Block List): Mostly dynamic ISP-customer addresses that shouldn’t be the source of SMTP.
3. XBL (Exploits Block List): IP addresses exhibiting the behaviour of infected hosts (worms, spambots, botnets etc)

Appearing on any of the above lists is generally bad news, and will result in many organisations blocking your email.

Removing Your IP

The removal process is straightforward; from spamhaus.org you can select the Blocklist Removal Center to confirm with an IP lookup that your IP is contained on any of the above lists.

If it is, a link is presented which will allow you to remove your IP. The link generates an email to your domain (I believe to both the email you provide manually and to postmaster@yourdomain.com), from which you must verify the request.

Once verified, you are warned that it can take “a maximum” of 30 minutes for the list to be updated. That may be so, but realistically you can add another 15-30 minute onto this before you’ll see the rejecting domains accepting your mail again.

Unhelpful Support

Of course, if you haven’t addressed the reason you were blocked in the first place, it’s only a matter of time before you are re-added to a Spamhaus blocklist. And this leads me to my main frustration with Spamhaus; it’s very difficult to get useful details on what kind of malicious traffic warranted your IP being added. You can forget about raw logs or even accurate timestamps; you will get a Spamhaus employee’s very general assumption on what might be causing the malicious traffic, a few impersonal and unexpectedly sinister “don’t let this happen again” remarks, and a few timestamps with an accuracy of +/- 30 minutes…yes, that’s not a typo, +/- 30 minutes!

Below is the extent of the help I received from the Spamhaus employee in tracking the source, just to demonstrate how vague and unhelpful their analysis is:

As it turned out, despite the lack of help from Spamhaus, finding the culprit wasn’t too difficult as we narrowed down the possibilities. It couldn’t be email egressing our email security appliance, as logs showed nothing unusual or malicious, and I was confident our SEG would have identified the malicious traffic anyway. A secondary SMTP server that we have almost fully migrated from similarly showed only legitimate mail being sent. A review of the firewall logs showed no SMTP traffic directly through the firewall from our internal LAN. However…we did find SMTP traffic successfully routing outbound from our WiFi, in particular a restricted Corporate SSID that has both internal and (as it turned out) unrestricted outbound access, NAT’d to the same public address as our email. D’oh!

The firewall logs clearly showed SMTP traffic being generated every few seconds from a mobile device, with a seemingly ever changing recipient IP (so this wasn’t some kind of device sending reports, which we do commonly see on our network, clinical refrigerators for example). A quick cross-reference of the MAC address from the Wireless LAN Controller against our mobile phone inventory, and the offending owner was found. As it turns out, the user had installed a free HD Movie App, which was essentially a P2P torrent app, generating all types of malicious traffic in the background. Once removed from the network, our IP stayed off the Spamhaus blocklist.

While emailing Spamhaus for support proved unhelpful, they do have some good general troubleshooting tips on their site, link below:

https://www.spamhaus.org/faq/section/Hacked…%20Here’s%20help

Prevention

So some takeaways and lessons from all this:

1. It’s not advisable to have both your email and general outbound traffic on the same NAT IP. Any malicious LAN traffic that manages to egress via this IP will negatively impact the reputation of your mail IP. This was something I had actually raised as a concern when replacing our email security gateway earlier this year, but we unfortunately took the advice of our vendor who didn’t consider it an issue.
2. No matter how locked down access to a network is, there is zero reason to allow general clients SMTP outbound. The WiFi network that this device was on was deemed securely locked down in terms of gaining access; strong WPA2 password, MAC filtering on the Wireless LAN Controller, and further MAC filtering on the DHCP scope. As such, at some stage a decision was made to leave this network with full outbound access. If a device needs SMTP outbound, it should be directed towards an authorised internal SMTP server.
3. Which leads to another failing on our part; allowing staff to install apps on company devices. We have for a long time intended to roll out MDM to have more control over our company devices, but there hasn’t been the desire to dedicate the time and resources to it. This will hopefully change that.
4. I’m disappointed that our Palo Alto firewall didn’t detect and take action against the sudden and fairly relentless SMTP spam this device was generating, so this will need to be reviewed, as will our firewall rules in general.