Grey on black: combining greylisting with RBL / blacklists

In the never-ending spam-fighting war, many different technologies are implemented to prevent users from wadingthrough a daily batch of pill discounts and lottery winnings. Two of them, which are both very effective, are greylists andRBLs (aka blacklists); they both operate at the SMTP level (even before the mail body is transferred). Usually, these technologies are deployed separately, but even better results can be achieved when combined. How? Follow me.

Greylisting

Greylisting is a very common anti-spam technology: whenever a potential spam arrives, our MTA replies with a temporary failure code (SMTP code: 4xx), and writes down the IP address of the sender MTA (the server that is sending the e-mail) in a DB; when facing a 4xx code, any RFC-compliant MTA will simply try to deliver the same e-mail again after a while, at which point our MTA will let the mail through (because the IP address is already present in the DB). On the other hand, most spammers do not follow the RFC (how bizarre): they would not bother retrying and would simply move on; this is because each retry is a memory/CPU cost for them (they need to note down the server which is temporary failing, keep a queue around, etc.); after all, if it was a real temporary failure, that server might still be failing for a few hours, and it would be a real waste of bandwidth to try again, just to stop at the same temporary failure.

What it is interesting about greylisting is that such a simple idea can achieve impressive results. If you google around, you will find enthusiastic people reporting at least 80% of spam blocked, with few of them citing numbers as high as 95%. All modern mail servers support greylisting either out of the box or through a plugin. For the famous open source server Postfix, the postgreyextension is easily available and widely deployed.

There obviously are some cons to greylisting:

  • Since normally greylisting is applied to all incoming e-mails, regular e-mail delivery is delayed. In fact, a RFC-compliant MTA will eventually retry, but how soon is an implementation detail. It might be one minute or one day (even though most servers usually retry in a few minutes). Obviously, mature greylisting implementations will keep the IP address of a good server in the DB for a long time (eg: one day or one week), effectively whitelisting it; so, the delay only happens on one e-mail per server. Most people don’t see this delaying as a problem, but I do: I believe that a good anti-spam pipeline should never impact regular e-mails whatsover. I don’t want a very important e-mail to be delayed for an unknown period of time because I cannot fight spam in other ways.
  • Some servers could be RFC-compliant and still fail to delivery e-mail to a server using greylisting. How & why? Simple: think of  large shop like gmail. They have dozens of SMTP servers. When a server tries to deliver an e-mail and it receives the 4xx code back, can you be absolutely sure that the sameSMTP server will retry later? What if they have an unique queue of e-mail, and another server picks the same e-mail up and retries? This other server will also be greylisted.  Can you see the problem now? I won’t enter details, but many greylist implementations suggest using manually-compiled (and maintained) whitelists. Uh-oh. Less funny now.

RBLs

RBLs are blacklists of servers/zombies that deliver spam, listed by IP address, and queried through the DNS protocol. In short, if you want to ask super-blacklist.example if 1.2.3.4 is listed there, it is sufficient to try to resolve the hostname 4.3.2.1.super-blacklist.example; if it resolves to an IP address (any IP address, it doesn’t matter), it’s listed; otherwise it is not listed.

Blacklists can be deployed within a mail server at different levels. For instance, you can use them in a score system to increase the probability for a mail to be treated as spam. Or you can use them to simply reject the e-mail at the SMTP level, even before the body is transferred (since everything you need to query the blacklist is the MTA IP address, and you have this information as soon as the TCP connection is initiated to your MTA).

Of course, not all blacklists are equal. If you rely on them for rejecting your e-mails, you better be sure that the blacklist is well-maintained, has a good reputation and that you are using it correctly. For instance, if we analyze the famous Spamhaus Zen blacklist, we find out that it is made of three different blacklists, one of which is the PBL. The PBL is (quoting) “a DNSBL database of end-user IP address ranges which should not be delivering unauthenticated SMTP email to any Internet mail server except those provided for specifically by an ISP for that customer’s use”. Rephrasing: it is a list of all dynamic IP ranges in the world; all home ADSL users are listed here. That’s it. There is no connection with spam altogether, it is just a list of dynamic IP addresses. Now, while most dynamic IP addresses will never deliver an e-mail directly to your server (and instead go through their ISP’s MTA), are you ready to block these hosts at the SMTP level? What if someone (one in ten thousands) runs a legitimate MTA server on their ADSL line? They are not violating any RFC, so why should you blacklist them? In fact, SpamHaus itself suggests against using this list for outright rejection of e-mail.

So, is there anything we can do with PBL at the SMTP level, which is in the middle between outright rejection and a SpamAssassin-like score system? Yes we can: we can greylist!

Greylisting & RBLs together

There are many different anti-spam technologies that can be done at the SMTP level, by looking at the so-called SMTP envelope (the SMTP conversation that happens before the actual e-mail is delivered). For instance, some people configure their servers so that they will reject e-mails based on the HELO command, eg: if its argument is not a FQDN hostname, as required by the RFC. Others (including myself) consider an outright rejection too risky, because there might be misconfigured MTAs out there and it’s not their users fault (nor yourusers fault!). Traditionally, you could either choose to implement one of these enforcements or not. But there is an alternative: greylisting.

Let’s get back to the PBL example. Using PBL for filtering e-mails is attracting:

  • Almost all regular e-mail will arrive from a non dynamic consumer IP address.
  • Almost all e-mails that you get from dynamic consumer IP addresses will be spam (basically, zombie Windows PCs running malwares that deliver spam).

But almost is not always, and I don’t want to risk rejecting any regular e-mail, even if it happens once a year. I don’t want to impact regular e-mails whatsover. So, you can do like me and just greylist entries from IPs that are in the PBL. The worst thing that can happen is that those e-mails are slighly delayed, but given the unlikeness of such event, it is probably an acceptable compromise.

If you think of it, this is just a smarter alternative way of deploying greylisting. Instead of greylisting each and every e-mail and causing delays to all your users, you greylist only those e-mails that are very dark grey but not really black. And if you google around for SMTP anti-spam tricks, you can find many of them which follows in this dark grey area: they’re so dark that some mail admins reject e-mail altogether, but they’re not fully black, so other more conservative admins just punt. Don’t punt: greylist!

This is why I greylist all e-mails from IPs in the PBL. And I also greylist all e-mails whose HELO string is not a FQDN, so that the occasionally misconfigured server is just delayed a little while, but not rejected, and I still block lots of spam very early in the pipeline.

And now, give me the code!

As much as I find this kind of greylisting deployment so sexy, it is depriving to see that there is basically no support around in mail servers for such a setup. The abandonware spftware called gl4rbl is the best I could find around, and we succesfully deployed an enhanced version of it with qmail in Develer for a long time.

When we switched to Postfix, we had to reimplement this, because the existing postgrey plugin does not allow conditional greylisting.

In the next days, we will post our own Postfix Policy Server that implements greylisting on RBL. It is a very compact Python script, less than 200 lines including comments. We also have some ideas on making it more generically useful so that it can be used to implement other kinds of conditional greylisting, and I will discuss them in the same post.

In case you did not know, you can avoid missing it by subscribing to the RSS feed.

UPDATE: the source code of pgl4rbl is now available on GitHub.