featured image, HTML format
Blog Farsight TXT Record

Getting More Results from DNSDB Using the New -O (Offset) Option

1. Introduction

In a recent Farsight blog article, three new features were announced, including “Incremental Result Transfers.” The article discussed the new features in the context of DNSDB Scout. In this article, I will focus specifically on how “Incremental Result Transfers” work if you’re using dnsdbq and it’s -O (“offset”) option. The motivation for this new feature is simple: Farsight knows that some users may want more than a million results. While you’re still limited to no more than a million results for any single query, you can now “dive deeper” by making additional queries “offset” into the pool of total results.

2. So How Deep Can I Dive?

The extent to which your account is able to request additional results can be seen with dnsdbq -I

$ dnsdbq -I
	reset: 2019-09-10 00:00:00
	limit: 1000
	remaining: 1000
	results_max: 1000000
	offset_max: 3000000

This quota report means that you can get up to FOUR million total results using the -O option in dnsdbq:

	Query		Offset			Limit (results/query)
				     Total:	4,000,000 results

If your account is NOT permitted to use the offset feature, you’ll see "N/A" for offset_max. For example, someone using a Farsight DNSDB Community Edition API key would see something like:

$ dnsdbq -I
	reset: n/a
	expires: 2019-10-09 13:47:47
	limit: 500
	remaining: 500
	results_max: 256
	offset_max: n/a
	burst rate: 25 per hour

3. A Worked Example

Request a million results for *.mit.edu domains in JSON Lines format using dnsdbq:

$ dnsdbq -r \*.mit.edu -l 1000000 -j > mit1.jsonl

Now let’s request a second tranche, offset by a million:

$ dnsdbq -r \*.mit.edu -l 1000000 -O 1000000 -j > mit2.jsonl

And a third tranche, offset by two million:

$ dnsdbq -r \*.mit.edu -l 1000000 -O 2000000 -j > mit3.jsonl

And add a fourth tranche, offset by three million:

$ dnsdbq -r \*.mit.edu -l 1000000 -O 3000000 -j > mit4.jsonl

Those four queries left us with:

$ wc -l mit[1234].jsonl
 1000000 mit1.jsonl
 1000000 mit2.jsonl
 1000000 mit3.jsonl
  146830 mit4.jsonl
 3146830 total

Now let’s combine and deduplicate those results…

$ cat mit[1234].jsonl | sort -u > mit-combo.jsonl

$ wc -l mit-combo.jsonl
  3146826 mit-combo.jsonl          <-- there were apparently 3146830-3146826=4 dupes

Note that we had four “dupes” even when we ran the four queries one right after the other.

4. Understanding the Limitations To The -O option to dnsdbq

Asking for all four tranches of results would require expenditure of four queries from your quota, not just one.

Farsight also cannot guarantee that there will be zero overlap between the sets of results returned since the underlying data may change while your queries are running as new data is received and incorporated. That is, we don’t “save” or “freeze” a copy of all results for each query you make, so we can’t strictly guarantee continuity of query results across tranches. If you think you may want more than one tranche full of results, we recommend you run all tranches in close succession for minimal overlap (like the four dupes in our example above).

If you’re going to sort and deduplicate your results (as shown in the example in the previous section), you’ll want to be sure to use JSON Lines format for output (e.g., dnsdbq's -j option) to ensure each observation is contained on a single line. Attempting to use traditional presentation format for output would “complicate” sorting since traditional presentation format spans multiple lines.

You should also note that the same RRname/RRtype Bailiwick/Rdata value may be seen multiple times in the output, perhaps with a count (or last-seen time) that has updated in the time between an initial run and a subsequent run.

5. Conclusion

If you’ve ever been frustrated when you have bumped into the max-million-results-per-query limit, the ability to now get up to four million results on most DNSDB API keys will be most-welcome news. We encourage you to explore this new feature and all the other new DNSDB API features!

The Farsight Security Sales Team can be reached at [email protected].

Joe St Sauver Ph.D. is a Distinguished Scientist with Farsight Security®, Inc.