Improved spam filtering: avoiding email forwarding services

Technical

In a recent blog post, I mentioned that one of the ways to improve spam filtering is to avoid using a forwarding service, and especially if you have your own domain, to point the MX records for your domain directly to our servers (Enhanced or family/business accounts only).

The reason for this is that FastMail does a lot of work at the SMTP stage (when email is transferred from an external system to FastMail) to try and identify spam bots and block them while letting legitimate through. Unfortunately many of these processes can only work when the MX records for your domain point directly to our servers. If you point the MX records to another system, and forward from that system to FastMail, we can't do the spam bot detection, and so more spam will get through.

To give you an idea of how effective this is, one of our users - the author, Simon Cann (http://simoncann.com/) - recently changed over the MX records for his Noise Sculpture domain (http://noisesculpture.com) to point to us instead of using a forwarding service. After doing so, he had the following to say:

As you suggested, I pointed the MX records for the Noise Sculpture
domain at FastMail and have been blown away by the results.  I'm now
getting maybe three or four pieces of spam in a day and the few pieces
of spam that are getting through are being correctly identified as
spam and put in the Junk Mail folder.

Simon went on to explain why he hadn't made the change sooner.

I came to FastMail a year ago following an unexpected and highly
inconvenient melt down of my web host who was also by default my email
host for my simoncann.com domain.  This was embarrassing and
potentially lost me business (I was in the middle of pitching a book
proposal to a group of literary agents).

On signing up with FastMail I immediately pointed the simoncann.com MX
record at FastMail.  My other website for my music-related books
(Noise Sculpture) was hosted by a different web host and so I
forwarded all of my email from that domain to FastMail.  I didn't see
a need to change the MX record--the web host was working well (still
is) and I couldn't be bothered to start hacking the DNS records.

Long story short, as the simoncann.com and noisesculpture.com emails
were sent to a single FastMail account, I hadn't realized how much
simoncann.com spam was getting killed by FastMail.  However, now that
I have changed the MX record for noisesculpture.com, I can see just
how much spam was getting through from that domain because I was using
forwarding rather than pointing the MX records.

This change has stunned me (in a very good way).  I'm so impressed
with the change (allied with your excellent service) that I've just
renewed my FastMail subscription to cover the next few years.

For the technically interested, the main things we do at the SMTP delivery stage to detect these spam bots are:

  • XBL blocking. The XBL is a
    highly accurate block list that lists the IPs of machines with known
    trojans and proxies. Independent testing shows that the ZEN RBL (of
    which XBL is a part) has a high block rate, with basically no false
    positives
  • Address enumeration detection. Spammers are always trying to
    find new addresses to send their spam to, and they do that by
    enumerating email addresses at hosts. We have a system that will
    automatically detect this behaviour, and block the spam bot from
    discovering any additional information about whether an address is
    valid or not
  • Greylisting. Greylisting is a process designed to detect poorly
    written sending email servers, of which almost all spam bots are.
    While greylisting is very useful, it is sometimes frowned upon
    because poorly implemented versions would delay email delivery. The
    FastMail implementation uses a number of
    tweaks

    to ensure that legitimate email is delivered with no delay, while
    spam email is delayed or stopped from delivery
  • RFC violations. When a server is transferring email from one
    system to another, there's a series of conventions they have to
    follow defined in the SMTP RFC
    2821
    . Again, spam bots are
    often poorly implemented, or try and use tricks to work around this
    conventions. By detecting some of the common violations, more spam
    bots can be detected and blocked
  • Custom spambot detection. Through our large and long term
    experience with running an email service, we've been able to closely
    analyse how spam bots try to deliver email to our servers. By using
    this data, we're able to pre-emptively detect spam bots early only
    after they connect, and block them from delivering their spam email
    at all