How Can I Benchmark Thee? Let Me Count The Ways
When evaluating or benchmarking providers of passive DNS service, we’ve seenpeople employ or describe a variety of different measures of interest,including:
- the number of responses received for test queries (more being better)
- the speed with which test queries get answered (faster being better)
- the usability of the passive DNS interface (streamlined, intuitive,adaptive, self-teaching, etc.)
- the breadth of DNS record types (
A,
AAAA
- ,
MX
- ,
TXT
- , etc.) that areavailable
- the ability to time fence, limit by bailiwick, etc.
- and many more factors.
Maximum Number of Returned Results
One factor that does not get the attention it perhaps deserves is the abilityof a passive DNS service to deliver results at scale. Scale MATTERS.
That is, if a given query results in a million hits, and you want or need thatmany, can you get them? Or is your output arbitrarily limited to just somesubset of all known results, such as perhaps the first thousand or tenthousand? If you’re limited to some low value, you may be facing big problems.
Incomplete Results? Complete Potential For Big Mistakes
If your passive DNS solution can’t give you a complete answer, what little youdo receive as an answer may result in serious errors.
For example, assume you only get a hundred results when you search for passiveDNS data about an IP address. All 100 of those domains might lookunquestionably bad, perhaps showing clear signs of being phishing-related, orall being DGA (domain generation algorithm)-related. As a result of thatincomplete evidence you block that IP address.
Once you’ve done so, however, you learn that while those 100 domains were infact bad, there were thousands of other innocent domains on that same IP, youjust didn’t get to see them — bummer for you, sorry about the collateraldamage.
Naturally, the converse is also a possibility: the first 100 domains you seefor an IP might look and be great, but you might have many others, unreporteddue to low limits on returned results, that get a “free pass” they really don’tdeserve. Sorry about those false negatives, bub.
Bottom line, you need the ability to get ALL the results needed for you to beable to make a fully informed choice. Partial results are like trying to safelydrive a car when the windshield is blacked out and you can only see out theside windows!
Bad Guys Attempting to “Go Stealth” By Overwhelming Fixed Limits
If the number of results your passive DNS solution returns is limited to a lownumber, some miscreants may be tempted to intentionally leverage this realityin an attempt to hide their bad behaviors from investigation with passive DNS.Specifically, if the bad guys have more base domains or more FQDNs than can bedisplayed by some passive DNS systems, they’ve effectively guaranteed thatthey’ve just gone at least partially “stealth” — they will have domains thatthey can use, but which investigators may not be able to see.
Standing Out From The Crowd — When You’re Trying To Hide?
This is not a riskless strategy on the bad guys’ part. While they may be ableto use a plethora of base domains or large numbers of randomized subdomains tooverload some passive DNS systems, that strategy is a very “noisy” one, and onethat’s easily detectable if you’ve got a non-output-limited view of passiveDNS.
In fact, you could even imagine a product that reports the domains with thelargest number of unique FQDNs seen per day, hour, or other period of time. Ifyou’re a bad guy trying to hide, you wouldn’t want to end up on such a list.
Not All Output Caps Are Bad: Output Limits As Basic Safety/Sanity Checks
We know that sometimes limits on output are deployed to “protect users fromthemselves.” Without them, it can be easy to self-DOS oneself.
For example, passive DNS implementations that only have web interfaces oftendon’t cope well with million-result responses: the user’s browser or systemcan easily become overwhelmed and become slow or crash. In that case, limitingqueries that might accidentally return an unexpectedly large number of resultsmay actually be a self-defense measure. Farsight, in fact, limits its ownDNSDB web interface for precisely that reason.
However, if you offer a command line interface (like Farsight’s own
dnsdb_query
), or an API like Farsight’s that can be directly integrated intoyour own custom code, those are the sort of options that can routinely copewith million record response, and that’s why Farsight allows users to adjustthe number of results that get returned, up to 1,000,000 per query for theAPI/CLI.
And obviously if you have Farsight’s DNSDB Export (“on premises”) product,the sky’s the limit.
Eliminating the Need For Million Record Responses
Farsight also gives you the tools you need to keep your output to manageablesize. At least in some cases, you may not need (or want) to see a millionresults.
Maybe you only want results for the last 90 days. Maybe you only want recordsof a particular type. When you’re using Farsight’s passive DNS system, you havethe flexibility to ensure that you don’t get responses that you don’t need orwant.
Conclusion
If you find yourself evaluating passive DNS systems, be sure you don’taccidentally overlook a potentially critical factor: the ability to get ALLthe results you may want and need.
Make sure your passive DNS system is mature enough to deliver ALL the resultsyour queries may generate, even if that’s a very large number. Be sure yourpassive DNS solution will scale to meet your analytical needs.
Joe St Sauver, Ph.D. is a Scientist with Farsight Security, Inc.
Heading
