Running Bulk Queries in DNSDB For A File of IP Addresses

I. Introduction

Most DNSDB API users are familiar with using a command line tool such as

dnsdbq

to query DNSDB, while others may prefer to use the Farsight DNSDB Scout(tm) browser plugin for Chrome or Firefox.

If you use

dnsdbq,

you know you can easily run simple IP address queries such as:

$ dnsdbq -i 104.244.14.108
;; record times: 2018-09-27 03:57:22 .. 2019-07-22 14:44:31
;; count: 4417
fsi.io. A 104.244.14.108

;; record times: 2018-09-27 11:26:14 .. 2019-07-22 15:51:30
;; count: 1538
www.fsi.io. A 104.244.14.108
​[etc]


However, sometimes you may have a file with many IP addresses you want to check in DNSDB.

For the purpose of this blog, we’ll assume that you’re working from a Unix-like system (either a Mac, a Linux box, or a Windows system running the Windows Subsystem for Linux (see https://www.farsightsecurity.com/blog/txt-record/WSL-20180315/ )).

II. Using A Shell File To Run Multiple dnsdbq queries

We’ll also assume that you’ve got a file containing a number of IPv4 (and/or potentially IPv6) addresses such as:

104.42.128.171
128.193.4.112
128.223.32.35
128.223.60.22
131.252.115.150
134.10.50.30
140.211.209.32
140.211.127.171
198.199.109.37
2620:105:b050:1550::90
2001:468:d01:20::80df:2023


You’d like to run those through DNSDB using

dnsdbq

. You could just edit your list of IP addresses to create a command file consisting of a series of

dnsdbq

commands such as:

$ cat run-some-ips.sh
dnsdbq -i 104.42.128.171
dnsdbq -i 128.193.4.112
dnsdbq -i 128.223.32.35
​[etc]


You’d then execute that command file by saying:

$ chmod a+rx run-some-ips.sh
$ sh run-some-ips.sh > run-some-ips.output
$ more run-some-ips.output
;; record times: 2018-05-02 11:00:50 .. 2019-07-22 13:22:10
;; count: 5507
qnamaker.ai. A 104.42.128.171

;; record times: 2018-03-09 05:08:54 .. 2018-11-13 09:35:57
;; count: 16
abma.asn.au. A 104.42.128.171
​[etc]


That works, but is somewhat “clunky” and inelegant.

III. The dnsdbq -f option

A more refined option is to use the

dnsdbq -f

option, which reads in and runs a file of DNSDB queries from

<stdin>

. That file must be specially formatted (perhaps with your favorite editor) to look like the following for IP queries:

$ cat my-ips.dat
rdata/ip/104.42.128.171
rdata/ip/128.193.4.112
rdata/ip/128.223.32.35
​[etc]


You could then run those queries by simply saying:

$ dnsdbq -f < my-ips.dat > my-ips.output
$ more my-ips.output
;; record times: 2018-05-02 11:00:50 .. 2019-07-22 13:22:10
;; count: 5507
qnamaker.ai. A 104.42.128.171

;; record times: 2018-03-09 05:08:54 .. 2018-11-13 09:35:57
;; count: 16
abma.asn.au. A 104.42.128.171
​[etc]


You’re not limited to running only IP queries using the

dnsdbq -f

option, although the format of the lines in your data file will be a little different.

If you were doing left hand side RRname queries instead of IP queries, your file of queries might look like:

rrset/name/www.uoregon.edu
rrset/name/www.oregonstate.edu
rrset/name/www.eou.edu
​[etc]


And if you were doing right hand side Rdata name queries, your file of queries might look like:

rdata/name/phloem.uoregon.edu
rdata/name/ns1.oregonstate.edu
rdata/name/mx.smtp.ucla.edu


You can also create an input file that’s a mixture of IP queries, RRset name queries, and Rdata name queries, if that happens to be convenient for you — just use the appropriate prefix for each line in the file.

We should also mention that you can also use other common

dnsdbq

options, such as asking for output in JSON lines format, or using time fencing, or specifying the maximum number of results to return for each query, should you desire to do so.

See the

dnsdbq

manual page for details about all

dnsdbq

options:

$ man dnsdbq
dnsdbq(1) BSD General Commands Manual dnsdbq(1)

NAME
dnsdbq -- DNSDB query tool

SYNOPSIS
dnsdbq [-dfgmhjsShcI] [-A timestamp] [-B timestamp] [-b bailiwick]
[-i ip] [-l query_limit] [-L output_limit] [-u server_sys]
[-n name] [-k sort_keys] [-P page] [-p output_type] [-R raw_rdata]
[-r rdata] [-t rrtype] [-J input_file]

DESCRIPTION
dnsdbq constructs and issues queries to the Farsight DNSDB and displays
responses. It is commonly used as a production command line interface to
the DNSDB API server.

DNSDB is a database that stores and indexes both the passive DNS data
available via Farsight Security's Security Information Exchange as well
​[etc]


IV. Quota Considerations

When doing bulk queries, please note that each query (whether aseparate dnsdbq query run as part of a script, or a line in a bulkdnsdbq -f run), will consume one query from the user’s quota.

If the user is using a free DNSDB Community Edition (“DCE”) key,burst rate limitations clamp the number of bulk queries you cando per hour to no more than 25. If you accidentally attempt to domore than 25, you’ll get a 429 error status with the text “Error:Burst window rate limit exceeded. (This is an intentionalconstraint associated with all DCE keys, but which doesn’t applyto regular DNSDB API keys)

V. Conclusion

You now know two approaches you can use to easily run a “file full” of queries in DNSDB with

dnsdbq

.

If you don’t currently have a DNSDB API key, but would like to try DNSDB, please see https://www.farsightsecurity.com/get-started/

I would like to thank Farsight Security staff David Waitzman and Jeremy Reed for their feedback on this article.

The Farsight Security Sales Team can be reached at [email protected].

Joe St Sauver Ph.D. is a Distinguished Scientist with Farsight Security®, Inc.