DomainTools Announces Predictive Threat Feeds - Powering Preemptive Exposure Management

Farsight TXT Record

Batch mode and raw query updates to dnsdbq

Written by:

Farsight Security

Published on:

Jan 13, 2020

On This Page

TOC Element

Introduction

For querying Farsight DNSDB, the world’s largest historical passive DNS database, many users like our Farsight DNSDB Scout browser plugins. Others prefer a command line tool such as

dnsdbq

, our evolving C language tool.

dnsdbq

is open source software, hosted on github.com, that you build yourself.

We recently made a number of small improvements to

dnsdbq,

which this blog article will describe. These updates correspond to version 1.5 of the software, which is tagged in github as

v1.5.0

There are no changes to the DNSDB API. To learn more about DNSDB API, visit here.

Raw queries

The DNSDB API allows making “raw” queries using a hex representation of encoded DNS information. In the previous version,

dnsdbq

had a command-line switch that supported making raw hex Rdata queries (aka the right-hand side). The API also supports raw RRset queries (aka the left-hand side). The raw Rdata queries used the

-R

switch, even though a non-raw Rdata query used the

-n

switch (for name). There was no switch available for raw RRset, but batch mode did support such queries.

In this release of

dnsdbq

, we have renamed the raw

-R

switch to

-N

, corresponding to the non-raw

-n

switch.

We added a new

-R

switch, corresponding to the non-raw

-r

switch.

To recap:

What Do I Want to Search? Was Now Command Line Example Left-Hand Side Regular -r -r (same) dnsdbq -r fsi.io Raw Hex N/A -R (new) dnsdbq -R 0366736902696f00 Right-Hand Side Regular -n -n (same) dnsdbq -n fsi.io Raw Hex -R -N (swapped) dnsdbq -N 0366736902696f00 Right-Hand Side IP -i -i (same) dnsdbq -i 4.0.0.0/8

We documented raw queries in 2016 in Raw Hex Rdata Queries: An Obscure (But Potentially Quite Useful) Bit Of DNSDB Functionality We wrote there that a raw query:

“is the ultimate search mode because it allows lookups for arbitrary octet strings (or prefixes of octet strings). That is, it allows you to bypass the DNS “presentation layer” and directly query the underlying data store. This works for existing RRtypes as well as unknown RRtypes, and in fact this is the only way to search the rdata of unknown RRtypes.”

Raw queries also allow querying for illegally formed DNS names. These might be formed by improperly punycoded international domain names (IDNs).

The raw encoding method is described on the dnstable-encoding manual page.

Enhanced batch mode output

If you’re running a big swath of queries in batch mode, it can sometimes be hard to tell what output corresponds to what query. We added an option for a enhanced (i.e. verbose) batch mode output to address that problem.

The existing output format from batch mode merely separated the results from different queries by a line with two dashes. For example, given input file

input.batch

containing:

rrset/name/farsightsecurity.com rdata/ip/4.0.0.0,8

The original (and continuing)

dnsdbq

behavior with one

-f

option is:

$ dnsdbq < input.batch -j -l 1 -f {"count":366,"time_first":1574284100,"time_last":1576847905,"rrname":"farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.","rdata":["104.244.14.108"]} -- {"count":4,"time_first":1574755050,"time_last":1576380492,"rrname":"4.xip.io.","rrtype":"A","rdata":"4.0.0.0"}

Now, if you specify two

-f

options, the outputs will be labeled with the original query after a

++

and the status of the query will be shown after the trailing

--

. You can specify two

-f

options as

-f -f

-ff

$ dnsdbq < input.batch -j -l 1 -f -f ++ rrset/name/farsightsecurity.com {"count":366,"time_first":1574284100,"time_last":1576847905,"rrname":"farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.","rdata":["104.244.14.108"]} -- NOERROR (no error) ++ rdata/ip/4.0.0.0,8 {"count":4,"time_first":1574755050,"time_last":1576380492,"rrname":"4.xip.io.","rrtype":"A","rdata":"4.0.0.0"} -- NOERROR (no error)

The two

-f

options feature is not (yet) compatible with the

-m

(merge) option. The merge option merges results from multiple batch mode queries together.

For a discussion of bulk use of batch mode, see our blog article Running Bulk Queries in DNSDB For A File of IP Addresses.

Bug fix: Handle missing newline in batch mode input file

dnsdbq

previously improperly parsed a batch mode input file that was missing a newline at the end of the file. It reported an error for that line. This has been fixed.

Old buggy behavior (where the

printf

command is sending a two line input stream with the last line not being newline terminated):

$ printf "rrset/name/bbn.com\nrrset/name/fsi.io" | dnsdbq -j -l 1 -f {"count":342,"time_first":1574283947,"time_last":1576837693,"rrname":"bbn.com.","rrtype":"A","bailiwick":"bbn.com.","rdata":["192.1.122.24"]} -- batch line too long: rrset/name/fsi.io

Fixed behavior:

You can also specify the command line options together and get more verbose batch output with command:

$ printf "rrset/name/bbn.com\nrrset/name/fsi.io" | dnsdbq -jffl 1 ++ rrset/name/bbn.com {"count":343,"time_first":1574451673,"time_last":1577013618,"rrname":"bbn.com.","rrtype":"A","bailiwick":"bbn.com.","rdata":["192.1.122.24"]} -- NOERROR (no error) ++ rrset/name/fsi.io {"count":396,"time_first":1574465819,"time_last":1577033230,"rrname":"fsi.io.","rrtype":"A","bailiwick":"fsi.io.","rdata":["104.244.14.108"]} -- NOERROR (no error)

Bug fix: allow a bare `.` path name in a query

Curl did not allow using a bare

.

in a query URL. If passed a URL containing ‘foo/./foo’, it would change the URL to ‘foo/foo’. To illustrate this, we will show a query for “com” in the root “.” bailiwick. The command

dnsdbq -r 'com/NS/.'

sends a query for URL https://api.dnsdb.info/lookup/rrset/name/com/NS but the query should be for URL https://api.dnsdb.info/lookup/rrset/name/com/NS/.

Curl as of version 7.42.0 supports a new

CURLOPT_PATH_AS_IS

option to tell it to pass the

.

as is. If

dnsdbq

is built against that version or later of curl, it will use the new option to correct the misbehavior.

Curl’s CURLOPT_PATH_AS_IS option is available on the command-line as the

--path-as-is

option. If you are generating your own queries using curl (without using

dnsdbq

), you should use the

--path-as-is

option yourself. For example, the following illustrates how curl, without the

--path-as-is

option, mangles the URL removing the bare “.” in “NS/.”:

$ curl -H 'Accept: application/json' -H "X-API-Key: $APIKEY" "https://api.dnsdb.info/lookup/rrset/name/com/NS/.?limit=2" -v 2>&1 | egrep 'GET /' > GET /lookup/rrset/name/com/NS/?limit=2 HTTP/1.1

Here is how the URL is passed properly by adding the

--path-as-is

option:

$ curl --path-as-is -H 'Accept: application/json' -H "X-API-Key: $APIKEY" "https://api.dnsdb.info/lookup/rrset/name/com/NS/.?limit=2" -v 2>&1 | egrep 'GET /' > GET /lookup/rrset/name/com/NS/.?limit=2 HTTP/1.1

Building dnsdbq on Unix-like systems

The

dnsdbq

README file describes how to install the few

dnsdbq

dependencies on a few Unix-like systems. After following those instructions, in a standard Unix-like environment you should be able to do the following to build, install and cleanup a

dnsdbq

installation:

$ sudo make all install clean

To test that it’s installed and see its version number:

$ dnsdbq -v dnsdbq version 1.5

Building dnsdbq on Windows

We have an earlier blog article Using Farsight’s dnsdbq Command Line DNSDB Tool in a Microsoft Windows Environment: The “Windows Subsystem for Linux” Option that also describes how to build and install it on Windows.

Conclusion

We hope this article has explained the recent changes to

dnsdbq

and given you some pointers to other resources about it. You’ve learned that:

you can now do raw hex RRset queries as well as raw hex Rdata queries,
the option naming for raw hex queries has changed to better parallel the regular (non-raw) query argument nomenclature,
that bulk batch mode now is more usable because output can be tagged with the corresponding query, and
treatment of the root bailiwick has been cleaned up.

For your reference, you can download a PDF version of the

dnsdbq

manual page here.

David Waitzman is a Senior Distributed Systems Engineer for Farsight Security, Inc..

No items found.

Farsight TXT Record

Farsight Security