DNSDBQ Technical Overview
dnsdbq is a command line tool that accesses the DNSDB passive DNS database via the Farsight Security DNSDB API server.
DNSDB is a database that stores and indexes both the passive DNS data available via Farsight Security’s Security Information Exchange as well as the authoritative DNS data that various zone operators make available. DNSDB makes it easy to search for individual DNS RRsets and provides additional metadata for search results such as first seen and last seen timestamps as well as the DNS bailiwick associated with an RRset.
Using dnsdbq
dnsdbq constructs and issues queries to the Farsight DNSDB and displays responses. It is commonly used as a production command line interface to the DNSDB API server.
Use of dnsdbq to access the Farsight DNSDB data requires a valid API key. To request a demonstration of DNSDB or to inquire about a trial API key please contact the DomainTools sales team.
dnsdbq can be used in two ways:
- Command line execution: all the query parameters are specified on its unix command line, it runs, outputs results and exits.
- Batch mode: some of the query parameters are specified on the command line, but it also reads an input batch file, which can be a pipe from another process. This process can keep the connection open and feed it multiple queries over time, with
dnsdbq emitting output for each one. See Batch mode and raw query updates to dnsdbq for more information about this.
This tool is written in the C programming language and released as open source. It can be downloaded from the dnsdbq GitHub repository.
Instructions for installing and getting started with dnsdbq can be found here https://github.com/dnsdb/dnsdbq/blob/master/README
Configuring dnsdbq for use
To configure dnsdbq for use, you need to place your API key into a configuration file on the system. You can as an alternative set the DNSDB_API_KEY environment variable as well.
$ echo 'DNSDB_API_KEY="yourDNSDB_API_KEY"' > ~/.dnsdb-query.conf
Verify that it’s correct.
$ cat ~/.dnsdb-query.conf
DNSDB_API_KEY="yourDNSDB_API_KEY"
Run a test query
Try running a test query.
$ dnsdbq -I
rate:
reset: 2021-07-05 00:00:00
limit: 1000
remaining: 1000
results_max: 1000000
offset_max: 3000000
If you get data back from the test query, dnsdbq is properly installed and configured and ready for use.
dnsdbq Command Line Options
OPTIONS
-a: enables ASINFO/CIDR annotation for IP addresses in A (IPv4 address) RRsets. Depending on the data source given by -D, the result might be of a least-specific, most-specific, or intermediate route.
-A timestamp: Specify a backward time fence. Only results seen by the passive DNS on or after this time will be selected. See also -c. See the TIMESTAMP FORMATS section for more information about this.
-B timestamp Specify a forward time fence. Only results seen by the passive DNS sensor network on or before this time will be selected. See also -c. See the TIMESTAMP FORMATS section for more information about this.
-b bailiwick: specify bailiwick (only valid with -r queries).
-c: by default, -A and -B (separately or together) will select partial overlaps of database tuples and time search criteria. To match only complete overlaps, add the -c (“completeness”) command line option (this is also known as “strict” mode).
-D: specify the data source for ASINFO/CIDR annotations, if enabled by -a. Default is asn.routeviews.org, but you may wish to try aspath.routeviews.org.
-d: enable debug mode. Repeat for more debug output.
-f: specify batch lookup mode allowing one or more queries to be performed. Queries will be read from standard input and are expected to be in one of the following formats:
- RRset (raw) query: rrset/name/NAME[/RRTYPE[/BAILIWICK]]
- RRset (raw) query: rrset/raw/HEX[/RRTYPE[/BAILIWICK]]
- Rdata (name) query: rdata/name/NAME[/RRTYPE]
- Rdata (IP address) query: rdata/ip/ADDR[,PFXLEN]
- Rdata (raw) query: rdata/raw/HEX[/RRTYPE]
- Change query options: $OPTIONS {options}
Where options are:
- -A timestamp
- -B timestamp
- -c
- -g
- -G
- -l query_limit
- -L output_limit
- -O offset
$OPTIONS alone on a line allows command line options to be changed mid-batch. If no options are given, the query parameters will be reset to those given on the command line, if any, or else to defaults.
A line starting with a # will be ignored as a comment.
Any internal slash (/) or comma (,) characters within the search names of a batch entry must be URL-encoded (for example, or ).
For raw queries, the HEX value is an even number of hexadecimal digits specifying a raw octet string. The “raw” wire-format encodings are standardized. The embedding of these in dnstable is documented in the dnstable-encoding(5) manual page.
In batch lookup mode, each answer will be followed by a — marker, so that programmatic users will know when it is safe to send the next lookup, or if lookups are pipelined, to know when one answer has ended and another begun. This option cannot be mixed with -n, -r, -R, or -i. See the EXAMPLES section for more information on how to use -f.
If two -f options are given, then each answer will also be preceded by a ++ marker giving the query string (as read from the batch input) in order to identify each answer when a very large batch input is given, and the — marker will include an error/noerror indicator and a short message describing the outcome. With two -f options and also -m, answers can appear in a different order than the batched questions.
The ++ and — markers are not valid JSON, CSV, or DNS (text) format, so caution is required. (See -m option below.)
-g: return graveled results. Default is to return aggregated results (rocks, vs. gravel). Gravel is a feature for providing Volume Across Time.
-G: undo the effect of -g, this returning rocks rather than gravel. (Used in $OPTIONS in batch files.)
-h: emit usage and quit.
-I: request information from the API server concerning the API key itself, which may include rate limit, query quota, query allowance, or privilege levels; the output format and content is dependent on the server_sys argument (see -u ) and upon the -p argument. -I -p json prints the raw info. -I -p text prints the information in a more understandable textual form, including converting any epoch integer times into UTC formatted times.
-i ip: specify rdata ip (“right-hand side”) query. The value is one of an IPv4 address, an IPv6 address, an IPv4 network with prefix length, an IPv4 address range, or an IPv6 network with prefix length. If a network lookup is being performed, the delimiter between network address and prefix length is a single comma (“,”) character rather than the usual slash (“/”) character to avoid clashing with the HTTP URI path name separator. See EXAMPLES section for more information about separator substitution rules.
-J input_file: opens input_file and reads newline-separated JSON objects therefrom, in preference to -f (batch mode) or query mode. This can be used to reprocess the output from a prior invocation which used -j (-p json). Sorting, limits, and time fences will work. Specification of a domain name, RRtype, Rdata, or offset is not supported at this time. If input_file is “-” then standard input (stdin) will be read.
-j: specify newline delimited json output mode.
-k sort_keys: when sorting with -s or -S, selects one or more comma separated sort keys, among “first”, “last”, “duration”, “count”, “name”, and/or “data”. The default order is be “first,last,duration,count,name,data” (if sorting is requested.) Names are sorted right to left (by TLD then 2LD etc). Data is sorted either by name if present, or else by numeric value (e.g., for A and AAAA RRsets.) Several -k options can be given after different -s and -S options, to sort in ascending order for some keys, descending for others.
-l query_limit: query for that limit’s number of responses. If specified as 0 then the DNSDB API server will return the maximum limit of results allowed. If -l, is not specified, then the query will not specify a limit, and the DNSDB API server may use its default limit.
-L output_limit: clamps the number of objects per response (under -[R|r|N|n|i|f]) or for all responses (under -[fm|ff|ffm]) output to -output_limit. If unset, and if batch and merge modes have not been selected with the -f and -m options, then the -L output limit defaults to the -l– limit’s value. Otherwise the default is no output limit.
-M max_count: for the summarize verb, stops summarizing when the count reaches that max_count, which must be a positive integer. The resulting total count may exceed max_count as it will include the entire count from the last rrset examined. The default is to not constrain the maximum count. The number of rrsets summarized is also limited by the query_limit.
-m: used only with -f, this causes multiple (up to ten) API queries to execute in parallel. In this mode there will be no “–” marker, and the combined output of all queries is what will be subject to sorting, if any. If two -f flags are specified with -m-, the output will not be merged, can appear in any order, will be sorted separately for each response, and will have normal ‘–‘ / ‘++’ markers. (See -f option above.)
-N HEX: specify raw -rdata data (“right-hand side”) query. HEX is as described above.
-n name: specify -rdata name (“right-hand side”) query. The value is a DNS domain name in presentation format, or a left-hand (“.example.com”) or right-hand (“www.example.”) wildcard domain name. Note that left-hand wildcard queries are somewhat more expensive than right-hand wildcard queries.
-O offset: to offset by #offset the results returned by the query. This gives you incremental results transfers. Cannot be negative. The default is 0.
-p output_type: select output type. Specify:
- text for presentation output meant to be human-readable. This is the default.
dns is a synonym, for compatibility with older programmatic callers.
- json for newline delimited JSON output.
- csv for comma separated value output. This format is information losing, since it cannot express multiple resource records that are in a single RRset. Instead, each resource record is expressed in a separate line of output. See the
DNSDBQ_TIME_FORMAT environment variable below for controlling how timestamps are formatted for this option.
-q: makes the program reticent about warnings.
-R HEX: specify raw rrset owner data (“left-hand side”) query. HEX is as described above.
-r rrset: specify rrset (“left-hand side”) name query.
-s: sort output in ascending key order. Limits (if any) specified by l and L will be applied before and after sorting, respectively. In batch mode, the -f, -ff, and -ffm option sets will cause each batch entry’s result to be sorted independently, whereas with -fm, all outputs will be combined before sorting. This means with -fm there will be no output until after the last batch entry has been processed, due to store and forward by the sort process.
-S: sort output in descending key order. See discussion for -s above.
-T transform[,…]: specify one or more transforms to be applied to the output:
- datefix: always show dates in human readable format (so, not in database format). This will be the format selected by the DNSDBQ_TIME_FORMAT environment variable, if set.
- reverse: show the DNS owner name (rrname) in TLD-first order (so, COM.EXAMPLE rather than EXAMPLE.COM).
- chomp: strip away the trailing dot (.) from the DNS owner name (rrname).
-t rrtype: specify the resource record type desired. Default is ANY. If present, this option should precede any -R, -r, -N-, or -n options. This option is not allowed if the -i option is present. Valid values include those defined in DNS RFCs, including ANY. A special-case supported in DNSDB is ANY-DNSSEC, which matches on CDS, CDNSKEY, DNSKEY, DS, DLV, NSEC, NSEC3, NSEC3PARAM, RRSIG, and TA resource record types.
-u server_sys: specifies the syntax of the RESTful URL, default is “dnsdb”.
-V verb: The verb to perform, i.e. the type of query, either “lookup” or “summarize”. The default is the “lookup” verb. As an option, you can specify the “summarize” verb, which gives you an estimate of result size. At-a-glance, it provides information on when a given domain name, IP address or other DNS asset was first-seen and last-seen by the global sensor network, as well as the total observation count.
-U: turns off TLS certificate verification (unsafe).
-v: report the version of dnsdbq and exit.
-4: use to force connecting to the DNSDB server via IPv4.
-6: use to force connecting to the DNSDB server via IPv6.
-8: Normally dnsdbq requires that -n or -r arguments are 7-bit ASCII clean. Non-ASCII values should be queried using PUNYCODE IDN encoding. This -8 option allows using arbitrary 8 bit values.
Rocks and Gravel
dnsdb data is stored in a set of time buckets. For every query, data is returned based on what is found in each time bucket. When in ‘rocks’ mode, this data is aggregated into a single returned set of data. In gravel mode, the data is returned unaggregated with records from each of those time buckets being returned.
first_seen, last_seen and time will return the aggregated results of the data found in the query (rocks) gravel returns all of that data unaggregated.
Timestamp formats
Timestamps may be one of following forms.
- Positive unsigned integer: in Unix epoch format.
- Negative unsigned integer: negative offset in seconds from now.
- YYYY-MM-DD [HH:MM:SS]: in absolute form, in UTC time, as DNSDB does its fencing using UTC time.
- %dw%dh%dm%ds: the relative form with explicit labels. Calculates offset from UTC time, as DNSDB does its fencing using UTC time.
When using batch mode with the second or forth cases, using relative times to now, the value for “now” is set when dnsdbq starts.
A few examples of how to use timefencing options.
# tuples ending after Aug 22, 2015 (midnight)
$ dnsdbq ... -A 2015-08-22
# tuples starting before Jan 22, 2013 (midnight)
$ dnsdbq ... -B 2013-01-22
# tuples starting or ending from 2015 (midnight to midnight)
$ dnsdbq ... -B 2016-01-01 -A 2015-01-01
# tuples ending after 2015-08-22 14:36:10
$ dnsdbq ... -A "2015-08-22 14:36:10"
# tuples ending within the last 60 minutes
$ dnsdbq ... -A "-3600"
# tuples ending after "just now"
$ date +%s
1485284066
$ dnsdbq ... -A 1485284066
# batch mode with only tuples ending within last 60 minutes,
# even if feeding inputs to dnsdbq in batch mode takes hours.
$ dnsdbq -f ... -A "-3600"
Sample Queries
A few examples of how to specify IP address information.
# specify a single IPv4 address
$ dnsdbq ... -i 128.223.32.35
# specify an IPv4 CIDR
$ dnsdbq ... -i 128.223.32.0/24
# specify a range of IPv4 addresses
$ dnsdbq ... -i 128.223.32.0-128.223.32.32
Perform an rrset query for a single A record for farsightsecurity.com. The output is serialized as JSON and is piped to the jq program (a command-line JSON processor) for pretty printing.
$ dnsdbq -r farsightsecurity.com/A -l 1 -j | jq .
{
"count": 6350,
"time_first": 1380123423,
"time_last": 1427869045,
"rrname": "farsightsecurity.com.",
"rrtype": "A",
"bailiwick": "farsightsecurity.com.",
"rdata": [
"66.160.140.81"
]
}
Perform a batched operation for a several different rrset and rdata queries. Output is again serialized as JSON and redirected to a file.
$ cat rrset/name/wikipedia.org rrset/name/dmoz.org rrset/raw/0366736902696f00/A \
rdata/name/pbs.org rdata/name/opb.org rdata/ip/198.35.26.96 \
rdata/ip/23.21.237.0,24 rdata/raw/0b763d73706631202d616c6c > batch.txt
$ dnsdbq -j -f < batch.txt > batch-output.json
$ head -1 batch-output.json | jq .
{
"count": 2411,
"zone_time_first": 1275401003,
"zone_time_last": 1484841664,
"rrname": "wikipedia.org.",
"rrtype": "NS",
"bailiwick": "org.",
"rdata": [
"ns0.wikimedia.org.",
"ns1.wikimedia.org.",
"ns2.wikimedia.org."
]
}
RRset, Rdata, RRtype and Bailiwick
There are a number of DNS terms used in this document that might not be familiar. Below are some definitions with links to more detailed information:
- Bailiwick: Data is considered “in bailiwick” if the DNS record being evaluated comes from a server that is known to be responsible for distributing information about that domain
- Rdata: Rdata refers to the right hand side of a record in a
DNS response. For instance, if you consider a record of
www.farsightsecurity.com IN A 66.160.140.81the Rdata refers to 66.160.140.81, or the IP address. Rdata can also refer to another FQDN (Fully Qualified Domain Name), such asinfo.farsightsecurity.com - RRset: Rdata refers to the left hand side of a record in a DNS response and contains the owner name, TTL, CLASS, TYPE and RDATA. An RRset contains the resource records with the same owner name, same class, same type, and usually the same TTL
- RRtype: This field in the DNS record defines what type of data is being returned. Common values include A (host address),
AAAA (IPv6 host address), CNAME (Canonical name) and NS (Nameserver)
Farsight’s passive DNS infrastructure performs a complex process of “bailiwick reconstruction” where an RRset’s position within the DNS hierarchy is reconstructed. This serves two purposes:
- Provide context of the location of a given DNS record within the DNS hierarchy
- Prevents “untrustworthy” records that are a result of intentional or unintentional cache poisoning attempts from being replicated by downstream consumers.
For example, given the fully qualified domain name www.dachshund.example.com, valid bailiwicks would be dachshund.example.com, example.com, and com.
DNSDB also has the ability to perform inverse or rdata searches.
For more information on these terms, please see: What is a Bailiwick?, RRset and Rdata Demystified, and List of DNS record types linked below.
Additional Information
- dnsdbq GitHub repository
- dnsdbq man page
- dnsdbflex GitHub Repository
- Batch mode and raw query updates to dnsdbq
- DNSDB Get Started Guide
- Farsight DNSDB Documentation
- DNSDB Scout: GUI alternative to dnsdbq
- Getting “Human” (ISO8601) Datetime Stamps From dnsdbq JSON lines-formatted Output
- What is a Bailiwick?
- RRset and Rdata Demystified
- Farsight’s DNSDB Time Fencing: A Post-Attack “Time Machine”
- List of DNS record types
- DNSDB API Technical Datasheet
- DNSDB Frequently Asked Questions
- 8 Common DNSDB “Pivots” for Threat Hunting
