NOD and Greylisting

Introduction

Farsight Security, Inc.’s Newly Observed Domains (NOD)(tm) feed allows sites toignore new domains for a few minutes — or even for up to 24 hours — until theInternet’s reputation providers have had a chance to render an opinion on thosenew domains.

Some spammers may attempt to “race NOD,” trying to use domains virtuallyinstantaneously, before Farsight can update NOD and before customers canretrieve those updates. Fortunately, there’s a technique that can mitigate thisspammer strategy: greylisting.

How Does Greylisting Work?

One of the easiest ways to understand greylisting is by comparing it withmore polar alternatives:

• Blacklisting (also called “block listing”) is an out-and-out negative result:in an email context, if your IP or domain name is blacklisted, your trafficwill not be accepted. An example of a popular block list is Spamhaus’
 Zen blocklist.
• Whitelisting is the polar opposite, a guaranteed “pass” for good traffic.In an email context, this normally means that you’re a highly trusted senderthat carefully follows industry consensus best practices,and as a result your traffic is explicitly “pre-approved.” One example of awidely-trusted whitelist is theSpamhaus Whitelist.

Greylisting is an automatic mail processing heuristic that falls “in between”those two extremes. Despite the name (“greylisting”), it doesn’t involvecreating a traditional globally maintained list of “grey” domains. Rather,each site programmatically constructs (and automatically expires) its ownlist.

In a nutshell, greylisting “temporarily rejects” email from new sources with a4xx status code, effectively saying “Sorry, we can’t accept this message rightnow, please try again later.” Most “real” mail transfer agents (MTAs) such asPostfix, Exim, etc., will re-queue messages and routinely attempt redeliveryseveral times. Most spambot mailers, however, cannot, or do not bother to doso. Using greylisting is thus an easy way to block a lot of spambot-transmittedunwanted email.

Mailers that do successfully retry and eventually manage to deliver non-spammyemail are often automatically added to a locally-maintained list that exemptsthat site from further greylisting, typically for a locally-determined periodof time. This is done to keep greylisting from impacting high-volume butsenders that work hard to control spam from their services.

A more in-depth discussion of greylisting can be found in RFC 6647, “Email Greylisting: An Applicability Statement for SMTP,”.

An often-mentioned greylisting implementation isPostgrey, used with Postfix. A list of othergreylisting implementations can be foundhere.

How Does Greylisting Relate to NOD?

Greylisting complements and enhances the effectiveness of NOD. It ensures thateven if spammers try to “race NOD,” that attempt will be futile. It representsa second layer of “backup protection,” andprotection-in-depth is a fundamental strategy that fosters cyber-security success.

Individual sites may want to experiment, trying greylisting first, then NOD(or vice versa), to see which ordering offers the best spam protection AND the“least impact to legitimate email traffic.”

“What Do You Mean by ‘least impact to legitimate email traffic’?”

NOD intentionally only targets brand-new domains. This means that NOD willnever impact well-established domain names. This makes it quite safe for allsites to potentially use.

Greylisting, however, is a heuristic that potentially applies to all domains,new, old, or in-between. As such, it has a greater potential for causing“collateral damage.”

We’ll just consider two examples of this potential collateral damage:

• While standards-compliant MTAs (such as Postfix and Exim) routinely (andcorrectly) handle greylisting, some “custom MTAs” may not.
• Greylisting software implementations normally mitigate this issue by

whitelisting known-good senders that are also known to have “flaky” customMTAs, but such lists are never perfect. Inevitably some obscurenon-standards-compliant flaky sites end up being overlooked. This means if youdecide to use greylisting, some legitimate mail may potentially end up gettingpermanently blocked, as if it were spam.

• Greylisting can also introduce potentially-irritating latency. The classicexample of this would be trying to use a “password reset” link on a seldom-usedsite: if you’re urgently trying to reset a totally-forgotten password, bumpinginto greylisting (and having to wait even fifteen minutes for a password resetmessage to be retried and eventually be delivered) can feel like an eternity.

We emphasize, however, that these are greylisting “corner cases.” Many siteshappily use greylisting and never run into any issues.

Conclusion

As the “spam wars” go on, temporarily blocking newly observed domains withFarsight Security’s NOD blocklist — and potentially complementing thatprotection with greylisting approaches — can result in excellent anti-spamprotection-in-depth.

Joe St. Sauver is a Distributed Research Scientist for Farsight Security, Inc.