I. Introduction
dnsdbq is Farsight’s popular command line client interface to Farsight DNSDB and other passive DNS systems. It is available in an easy-to-build-from-source code form here.
Today’s blogpost describes some recent enhancements that have been made to that client by two of its authors, Paul Vixie and David Waitzman.
II. ASN Tagging
We’re happy to announce that the current version of dnsdbq, version 2.5.3, now supports ASN tagging.
Unless you have a network engineering background, you may not know what an ASN is, or how it can be useful to you when combating online abuse, so let’s start at the very beginning.
“ASN” stands for “Autonomous System Number” and, according to ARIN resource guide, is defined as:
“An Autonomous System (AS) is a group of one or more IP prefixes (lists of IP addresses accessible on a network) run by one or more network operators that maintain a single, clearly-defined routing policy.
“Network operators need Autonomous System Numbers (ASNs) to control routing within their networks and to exchange routing information with other Internet Service Providers (ISPs).”
However, ASNs can also help analysts tag/understand traffic. When an analyst is given a long list of IPs (or a long set of DNSDB results), ASN tagging can help sort out what’s going on where.
The easiest way to understand how ASN tagging works/what it looks like is with an example:
$ dnsdbq -r www.internet2.edu/A -a -S -k first
;; record times: 2020-11-11 13:16:17 .. 2021-02-23 09:32:37 (~103d 20h 16m)
;; count: 9954; bailiwick: internet2.edu.
www.internet2.edu. A 34.224.139.87 ; AS14618 34.224.0.0/12
;; record times: 2020-10-21 12:06:55 .. 2020-11-11 07:34:46 (~20d 19h 27m)
;; count: 1627; bailiwick: internet2.edu.
www.internet2.edu. A 52.71.134.14 ; AS14618 52.70.0.0/15
;; record times: 2010-12-28 02:52:57 .. 2012-02-03 05:45:52 (~1y ~37d)
;; count: 437013; bailiwick: internet2.edu.
www.internet2.edu. A 207.75.164.151 ; AS237 207.72.0.0/14
[etc]
When the dash lowercase a option is present, output gets enhanced with current IPv4-to-ASN information from the University of Oregon Route Views project, including:
- The originating Autonomous System Number (ASN), and
- The origin (and CIDR length) of the associated encompassing prefix.
Some notes on interpreting that enhanced output:
- If you’d like to try the new dash “a” option, be sure to update to the new master branch from Github.
- Be sure to also update your ~/.dnsdbq-query.conf file to use the latest version of the DNSDB API, version 2:
DNSDB_SERVER="https://api.dnsdb.info"
APIKEY="your_API_key_goes_here"DNSDBQ_SYSTEM="dnsdb2"
$ whois as393667
- You may also find it useful to lookup ASNs at Hurricane Electric’s terrific BGP site at https://bgp.he.net/. The site includes information on each ASNs prefixes, the ASN’s routing adjacencies, and much more.
- Important caveat: IPs get tagged with CURRENT ASN information — years ago things may have been different (e.g., some IP address space may have changed hands over time). This feature is obviously targeting relatively current activity, not ancient data!
- asn.routeviews.org is the default service used by dnsdbq for ASN tagging, but you can try the -D option to specify other services (such as aspath.routeviews.org, origin.asn.cymru.com, etc.). Why might you use one of these alternatives? Well, for example, aspath.routeviews.org actually shows you a sample route path (rather than just the origin ASN):
$ dnsdbq -r www.internet2.edu/A -a -S -k first -D aspath.routeviews.org
;; record times: 2020-11-11 13:16:17 .. 2021-03-22 12:39:37 (~130d 23h 23m)
;; count: 12972; bailiwick: internet2.edu.
www.internet2.edu. A 34.224.139.87 ; AS49788 AS174 AS16509 AS14618 34.224.0.0/12
;; record times: 2020-10-21 12:06:55 .. 2020-11-11 07:34:46 (~20d 19h 27m)
;; count: 1627; bailiwick: internet2.edu.
www.internet2.edu. A 52.71.134.14 ; AS49788 AS174 AS16509 AS14618 52.70.0.0/15
;; record times: 2010-12-28 02:52:57 .. 2012-02-03 05:45:52 (~1y ~37d)
;; count: 437013; bailiwick: internet2.edu.
www.internet2.edu. A 207.75.164.151 ; AS49788 AS174 AS7018 AS237 AS237 AS237 207.72.0.0/14
[etc]
III. “What dnsdbq Output Formats Support ASN Tagging?”
ASN tagging is offered in all output formats, not just the default presentation mode which you’ve already seen.
For example, this is what ASN-tagged dnsdbq JSON Lines format output looks like when “pretty printed” with jq.
$ dnsdbq -r www.internet2.edu/A -a -S -k first -j | jq '.'
{
"count": 11324,
"time_first": 1605100577,
"time_last": 1615214243,
"rrname": "www.internet2.edu.",
"rrtype": "A",
"bailiwick": "internet2.edu.",
"rdata": [
"34.224.139.87"
],
"dnsdbq_rdata": {
"34.224.139.87": {
"asinfo": {
"as": [
14618
],
"cidr": "34.224.0.0/12"
}
}
}
}
[etc]
ASN-tagged CSV format output looks like the following (lines wrapped for display here, and blank lines added for ease-of-reading; normally CSV output will be one line per observation, uncolorized, and with no blanks between lines):
$ dnsdbq -r www.internet2.edu/A -a -S -k first -p csv
time_first,time_last,zone_first,zone_last,count,bailiwick,rrname,rrtype,rdata,asnum,cidr
"2020-11-11 13:16:17","2021-03-08,14:37:23",,,11324,"internet2.edu.",
"www.internet2.edu.","A","34.224.139.87","14618","34.224.0.0/12"
"2020-10-21 12:06:55","2020-11-11 07:34:46",,,1627,"internet2.edu.",
"www.internet2.edu.","A","52.71.134.14","14618","52.70.0.0/15"
"2010-12-28 02:52:57","2012-02-03 05:45:52",,,437013,"internet2.edu.",
"www.internet2.edu.","A","207.75.164.151","237","207.72.0.0/14"
[etc]
IV. New Output Transforms: datefix
The latest version of dnsdbq also brings some new output transformation options:
-T transform[,...]
specify one or more transforms to be applied to the output:
datefix always show dates in human readable format (so, not in
database format). This will be the format selected by the
DNSDBQ_TIME_FORMAT environment variable, if set.
reverse show the DNS owner name (rrname) in TLD-first order (so,
COM.EXAMPLE rather than EXAMPLE.COM).
chomp strip away the trailing dot (.) from the DNS owner name
(rrname).
The nice thing about these new features is that they handle some of the things that analysts previously had to “fix” via user-supplied routines. This simplifies learning and teaching analysts to use dnsdbq.
Let’s look at these new options now.
You can use any of the options individually, or you can combine them.
The case for “datefix” is easily made if you’re an analyst who like JSON Lines format output. Normally that output looks like:
$ dnsdbq -r www.whitman.edu -j
{"count":1401263,"time_first":1395869220,"time_last":1616421069,"rrname":"www.whitman.edu.","rrtype":"A","bailiwick":"whitman.edu.","rdata":["199.89.174.11"]}
{"count":940920,"time_first":1277387381,"time_last":1395938425,"rrname":"www.whitman.edu.","rrtype":"A","bailiwick":"whitman.edu.","rdata":["199.89.174.13"]}
If you’re going to do further automated processing of those results, Un*x ticks are a natural format, but if you’re a human just trying to eyeball that data, “human times” will be far more convenient.
You have a choice of two formats for “human time.” The default format is “iso”. It can also be explicitly selected with:
$ export DNSDBQ_TIME_FORMAT="iso"
“iso” format human time output looks like:
$ dnsdbq -r www.whitman.edu -j -Tdatefix
{"count":1401263,"time_first":"2014-03-26T21:27:00Z","time_last":"2021-03-22T13:51:09Z","rrname":"www.whitman.edu.","rrtype":"A","bailiwick":"whitman.edu.","rdata":["199.89.174.11"]}
{"count":940920,"time_first":"2010-06-24T13:49:41Z","time_last":"2014-03-27T16:40:25Z","rrname":"www.whitman.edu.","rrtype":"A","bailiwick":"whitman.edu.","rdata":["199.89.174.13"]}
The alternative format, “csv” format times, eliminates the T between the date and the time, and also drops the trailing “UTC timezone” “Z”. Explicitly select it with:
$ export DNSDBQ_TIME_FORMAT="csv"
Now when you run dnsdbq queries with -Tdatefix, your “human time format” output will look like:
$ dnsdbq -r www.whitman.edu -j -Tdatefix
{"count":1401263,"time_first":"2014-03-26 21:27:00","time_last":"2021-03-22 13:51:09","rrname":"www.whitman.edu.","rrtype":"A","bailiwick":"whitman.edu.","rdata":["199.89.174.11"]}
{"count":940920,"time_first":"2010-06-24 13:49:41","time_last":"2014-03-27 16:40:25","rrname":"www.whitman.edu.","rrtype":"A","bailiwick":"whitman.edu.","rdata":["199.89.174.13"]}
The setting you specify with export will last only for the duration of your current session unless you add that setting to the appropriate dot file (~/.bash_profile or the equivalent for other Un*x shell environments).
V. New Output Transforms: reverse and chomp
The case for the other transforms is also easily articulated. Many times when you’re generating substantial volumes of output with dnsdbq, you may want to reverse the RRnames to make it easy to group related names together. Doing that previously required using a helper application, but now you can handle this transformation right from within dnsdbq.
For example, assume you’re looking at “A” records from the last 90 days for school districts under the k12.or.us effective TLD.
You’d like to see all the hosts for albany.k12.or.us in one clump, all the ashland.k12.or.us hosts in another clump, all the hosts for bandon.k12.or.us in a third clump, etc. The new Treverse,chomp option (often used in conjunction with dnsdbq’s well established “sort by name” option (-s -k name)) makes that easy:
$ dnsdbq -r \*.k12.or.us/A -Treverse,chomp -s -k name -A90d -l0 | more
-Treverse reverses the RRnames; -Tchomp "eats" the trailing dot from the RRnames:
;; record times: 2010-09-17 20:08:32 .. 2021-02-10 19:58:54 (~10y ~148d)
;; count: 993; bailiwick: bandon.k12.or.us.us.or.k12.bandon.mail A 198.237.212.67
[...]
;; record times: 2020-12-02 00:52:47 .. 2021-03-22 02:11:12 (~110d 1h 18m)
;; count: 524; bailiwick: bandon.k12.or.us.us.or.k12.bandon.www A 107.180.54.183
[...]
;; record times: 2010-07-08 13:15:51 .. 2021-03-22 09:41:39 (~10y ~259d)
;; count: 47324; bailiwick: canby.k12.or.us.us.or.k12.canby.blogs A 198.236.6.75
[etc]
VI. Conclusion
We hope you find these new features in dnsdbq to be useful. We think they’re going to make at least some analyses a lot easier to perform.
If you need information about arranging for a DNSDB API key to use with dnsdbq, please visit here.
Joe St Sauver is a Distinguished Scientist and Director of Research for Farsight Security, Inc..
Heading
