Avoiding the SPAM folder – SPF Records

At HelpEncourage.Me, I use email for a variety of purposes: invitations, reminders, resetting passwords, etc.  However, I noticed that the emails would often get moved into the SPAM folder by various email providers.

Obviously, that is not a good thing, especially since these are transactional emails that are triggered by a direct request from users.

Now, there are a variety of companies that will gladly help you navigate the treacherous waters of email deliverability, but integrating a third-party solution is still a ways down the pipeline.

At the time, it seemed like a valid trade-off between integration time and getting something launched.  Of course, that was what I thought before I went down the rabbit hole…

Where is my password reset email?!

I first noticed an issue with sending emails when I realized that I had forgotten my password on my account.  After upgrading and launching, I migrated all the old data, including a user account I had used during some alpha testing of the site.

Fortunately, I knew there was an easy to use password reset tool, so I filled in my account email, and waited for my reset token to arrive.

After 30 minutes or so, I began to suspect foul play.  Sure enough, when I went to my GMail SPAM folder, my reset email was tucked away with various offers for cheap online medications.

Not Good

Obviously having these emails end up in the SPAM folder is not a good thing.  I needed to do some research.  After an hour or so of researching, I found three common suggestions to help keep emails out of the SPAM folder.  The most obvious suggestion was to avoid spam-like language and topics.  Fortunately, my password reset email was pretty standard and didn’t fit the usual definition of “spamminess”, so I felt pretty safe there (I would verify this, but more on that later).

The next suggestion, DKIM (DomainKeys Identified Mail), was much more complicated and involved creating a public/private key pair, and using the private key to sign all of our emails so they could be verified using the public key.  Since this involved a code change, I decided to grab the low-hanging fruit first.

Enter SPF

SPF, or Sender Policy Framework, is basically a mechanism that ensures that the server sending a message is authorized to send email for the given domain.  It sounds very complicated, especially when you visit the SPF website.  Fortunately, it isn’t as difficult as it sounds, and there are a number of tools that can help you ensure that you have correctly created an SPF record for your domain.

Determining if SPF is enabled

The first step to implementing SPF for your domain is to check if it is already setup.  Sounds obvious right?  When it comes to email, I’ve learned not to take anything for granted.

If you have a GMail account, a simple way to check whether SPF is correctly enabled on your domain is to look at the headers for an email sent from the domain.  You can view the headers by selecting the “Show original” option for the email.  In the header you should find an “Authentication-Results” section.  The Authentication-Result section from my password reset email looked like the following:

Authentication-Results: mx.google.com; spf=neutral (google.com: 198.58.81.208 is neither permitted nor denied by best guess record for domain of request@helpencourage.me) smtp.mail=request@helpencourage.me

Oops.

You can see that my SPF was neutral, meaning that I haven’t specified anything about who can and can’t send emails from the HelpEncourage.Me domain.  Clearly, I needed to fix that.

Adding an SPF record to DNS

An SPF record is just a TXT record that starts with “v=spf1”, followed by a list of options that authorize and deny various servers from sending mail for the given domain.  The openspf.org website has a page describing the syntax and options available.

One of the problems that I encountered when testing various permutations of the SPF record is that some tools do no like the “-all” option (which, when included at the end of the record, indicates a “hard fail” for any server not explicitly allowed by the SPF record).  By changing the record to use the “soft fail” operator with the option (“~all” – note the tilde operator, not a minus), I was able to pass all the different verification tools available.

Creating the list of “authorized” servers took a bit of trial and error.  Initially, I authorized just the IP addresses identified by the A and MX records for my domain:

v=spf1 a mx ~all

I quickly realized (thanks once again to the informative headers provided by GMail), that my A and MX records were not enough as the actual IP address used for outgoing mail was not included.  So I manually added the IP address identified via gmail:

v=spf1 a mx ip4:198.58.81.208 ~all

That was all it took.  I waited a few hours for my DNS changes to propagate, and we were in business.

Tools

While setting up the SPF record seems easy now that I have it working, trying to troubleshoot problems was a pain.  Fortunately, there are a number of excellent resources that can help you identify any problems with your record and syntax.

The first place to go to check on things like syntax is the OpenSPF tools page.  Specifically, the Kitterman validation link and the SPF Explanation Tool are both hugely valuable resources.

Next, if you have a gmail account, Google puts all sort of useful information in their headers.  This is especially useful when you need the IP address of the actual outgoing mail server.

Finally, Port25 offers a free email address:

check-auth@verifier.port25.com

which you can use to check all sorts of things, including your SPF record, DKIM signing, and a message’s SPAM assassin score.  The SPAM assassin score is especially useful for avoiding the “spammy” content that can cause a message to get flagged, and is how I verified that my password reset email was not being tagged for its content.

All that is required is to send an email to that address and in a few minutes you will get back a reply with a wealth of information about the email and your setup.