I. Introduction
If you’re doing a targeted investigation, or doing a review of a small ISP’s limited network address space, it’s easy enough to manually check a small number of network prefixes (“CIDR blocks”) in DNSDB.
However, doing a larger investigations involving hundreds or even thousands of prefixes can quickly become tedious if done one-at-a-time. You could obviously script those queries, but in May 2017, we demonstrated a shortcut for situations where all the prefixes of interest were in the same Autonomous System Number (ASN) — we provided a little Scala demo application that would:
- Let you enter an Autonomous System Number of interest (such as AS3582, the University of Oregon’s ASN)
- Automatically retrieve a list of the prefixes originated by that ASN, and
- Look up all those prefixes in DNSDB, returning the associated domain names.
See Automatically Going From An Autonomous System Number (ASN), To The Prefixes It Announces, To The Domain Names Seen Using Those IPs: Another Farsight DNSDB API Demo Application and Whitepaper pointing at Checking DNSDB By ASN (ASN –> Prefixes –> Domain Names) Another Demonstration Scala Project
The interface from that project looked like this:
Figure 1. Sample Query DNSDB By ASN Application In Scala
To map ASNs to the prefixes they announced, that application relied on the BGP prefix data that was available from bgpview.io’s API. As can happen to any online service, bgpview.io was offline for a period of time, although we’re delighted to see they’re back as of July 25, 2018 as announced at https://twitter.com/bgpview.
Given that outage, though, we thought it might be worthwhile to show how the “Query-by-ASN” application could be adapted to other cloud sources of BGP prefix data. For example, we can try using RIPE NCC’s https://stat.ripe.net service.
II. Integrating RIPE’s stat.ripe.net Service With Our Application
Let’s take a quick look at the data we can get for this purpose from RIPE. On a Mac in a terminal window, we can use curl to retrieve routing data for an ASN of interest from the RIPE API server. We’ll stick that info into a file:
$ curl https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS3582 > 3582.txt
$ cat 3582.txt
{
"status": "ok",
"server_id": "stat-app8",
"status_code": 200,
"version": "1.2",
"cached": false,
"see_also": [],
"time": "2018-06-26T18:04:23.679067",
"messages": [
[
"info",
"Results exclude routes with very low visibility (less than 3 RIS full-feed peers seeing)."
]
],
"data_call_status": "supported - connecting to ursa",
"process_time": 68,
"build_version": "2018.6.20.351",
"query_id": "20180626180423-0cb31df8-7675-4a0b-9e57-d1fca7ddd1e8",
"data": {
"resource": "3582",
"prefixes": [
{
"timelines": [
{
"endtime": "2018-06-26T08:00:00",
"starttime": "2018-06-12T08:00:00"
}
],
"prefix": "2607:8400::/32"
},
{
"timelines": [
{
"endtime": "2018-06-26T08:00:00",
"starttime": "2018-06-12T08:00:00"
}
],
"prefix": "128.223.0.0/16"
},
[etc]
Obviously, a LOT of JSON format data gets returned by that one API call! In the interest of brevity, we limited what we’ve shown here — the bits we need are just the IPv4 and IPv6 prefixes (such as the ones shown above highlighted in red bold text).
At the command line, we can ask for just those prefixes from the RIPE data by using jq:
$ cat 3582.txt | jq -r '.data.prefixes[].prefix' 2001:468:d01::/48 128.223.0.0/16 2607:8400::/32 207.98.72.0/21 184.171.0.0/17
But how to easily get that data integrated within our Scala code? We decided to try using the json4s-jackson routines for this task (see http://json4s.org/ and https://mvnrepository.com/artifact/org.json4s/json4s-jackson ). Focusing on just the most relevant code from the updated application, we’re now doing:
var checkURL: String = "https://stat.ripe.net/data/announced-prefixes/data.json?resource=" + myasn rawJsonLines = scala.io.Source.fromURL(checkURL).mkString var json = (parse(rawJsonLines) \ "data" \ "prefixes" \ "prefix") var jsonArrayTemp = ((json).extract[Array[String]])
The above Scala commands:
- Define the URL we need to retrieve the prefix data from stat.ripe.net in JSON format
- Actually retrieve the required prefix data from RIPE, concatenating the multiple lines into a single long JSON blob
- Extract just the prefix data we need from the JSON blob, and
- Convert those prefixes into an array of prefix strings for further processing.
We then sort that array and ensure that there are no duplicate elements in the array by saying:
var jsonArray = jsonArrayTemp.sorted jsonArrayTemp = jsonArray.distinct jsonArray = jsonArrayTemp
At that point, we can loop over that array and query DNSDB for each unique prefix returned from the RIPE service, first processing IPv4 prefixes and then processing IPv6 prefixes. If curious, see the updated source code as described in Section III of this blog.
While we were updating the code, we also took the opportunity to do a little “clean up” of the record types offered in the interface. We’d originally offered users a choice of essentially all well-known record types, but that didn’t really make sense. To understand why, remember that we’re retrieving a list of IPv4 or IPv6 prefixes, and then looking up those prefixes in DNSDB’s Rdata. The only relevant record types for that Rdata query would be “A” records (for IPv4 addresses), “AAAA” records (for IPv6 addresses), and “ANY” (to get either IPv4 or IPv6 addresses). We’ve removed the others to simplify the code and to avoid potentially confusing users.
III. Retrieving and Installing The Updated Application
We assume that you’re working on a Mac running High Sierra. We also assume that your Mac is fully patched up-to-date and has been recently backed up — if that’s not true, please update, patch and backup your system before proceeding. Next:
Confirm that you have a current DNSDB API key; without a DNSDB API key, you won’t be able to use this application. Visit here if you need to arrange for a trial DNSDB API key.
That key should be put in a file called .dnsdb-apikey.txt (NOTE the leading dot!), in your home directory. The file must contain ONLY your 64-character DNSDB API key and nothing else (no quotes, no “comments,” etc.). It MUST be a plain text file (not a MS Word document or other formatted file).
Be sure you’ve already installed Apple’s developer’s tools, Xcode, including the command line tools. If you don’t already have Xcode installed, visit here.
For the installation of prepackaged libraries, we’ll rely on the Brew package manager. If you don’t already have Brew installed, visit here.
We also need the Java SDK. There are a confusingly large number of different versions of Java, so to keep this simple, let me simply say that anyone who wants to try this application should install Java SE Development Kit 8u172 or later. See http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html (scroll down to see this version). Please install this version even if you’ve already got some other version of Java installed.
Also install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files. The two jar files you need are bundled by Oracle into a single zip archive file available from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
Once you’ve download that archive and you’ve unzipped the archive, copy the unzipped jar files into the lib/security directory that’s under your Java Runtime’s home directory. If you have trouble successfully doing this, see page 22-24 and Appendix VI of this whitepaper
Once you’ve gotten all the above prerequisites accomplished (whew!), it’s then time to install Scala and sbt. Scala and sbt are easy to install on the Mac using brew:
$ brew update $ brew install scala $ brew install sbt
Scala is a highly extensible language. While it may seem like we’ve already downloaded a “lot” of software, Scala and sbt will typically download still more (as our choice of Scala features may require). Most often, additional required libraries will be automatically retrieved from the Maven repository by sbt.
A copy of the complete updated and adapted application source is available to be downloaded at https://dl.farsightsecurity.com/dist/blog/scaladnsdb3/check-asn2-distrib.tar.gz or from https://github.com/farsightsec/blog-code/tree/master/scaladnsdb3.
Download that file and ensure it untars into a directory called CheckASN2
$ tar xfv check-asn2-distrib.tar.gz x CheckASN2/ x CheckASN2/project/ x CheckASN2/project/assembly.sbt x CheckASN2/project/build.properties x CheckASN2/project/license.sbt x CheckASN2/project/plugins.sbt x CheckASN2/lib/ x CheckASN2/build.sbt x CheckASN2/src/ x CheckASN2/src/test/ x CheckASN2/src/test/resources/ x CheckASN2/src/test/java/ x CheckASN2/src/test/scala/ x CheckASN2/src/windows/ x CheckASN2/src/windows/License.rtf x CheckASN2/src/main/ x CheckASN2/src/main/scala/ x CheckASN2/src/main/scala/CheckASN.scala x CheckASN2/src/main/scala/CheckRemainingQuota.scala x CheckASN2/src/main/scala/CheckAPI.scala x CheckASN2/src/main/scala/UIerror.scala x CheckASN2/src/main/scala/UIerror2.scala
Change down into that directory and run the Scala application by saying:
$ cd CheckASN2 $ sbt run
The application should then run. The first time it executes, it may take a minute to download all the resources it requires. Please be patient, this should only happen once. You should then be up and running.
Try using the application to investigate an ASN of interest.
For help identifying a relevant ASN, visit https://bgp.he.net/ and use the search box on that website to search for an organization of interest. For example, you could search for NERO. That’s the Network for Education and Research in Oregon, AS3701.
Caution Before Proceeding: When you do a query-by-ASN run, each prefix retrieved from DNSDB counts as a “query.” If you have a limited number of queries available and you ask for a large ASN with many prefixes (such as Verizon’s AS701 with many, many prefixes!), you may exhaust your remaining quota with just one or two runs!
Plug the ASN number that’s of interested into the updated Scala application and hit Search at the bottom of the interface. Note that the Scala application’s “Search” button will stay colored blue until it finishes running.
Your output will be in one or more text files under a folder called dnsdb-output in your home directory. Look under today’s date, and then in a subdirectory tagged with the time and ASN you searched for… For example:
Figure 2. Output from a Query-by-ASN Run for AS3701
Note: What would normally be a slash in a CIDR prefix is replaced with a comma, and any colons in IPv6 prefixes are replaced with pound signs.
You can open any of the text files from a Query-by-ASN run just as you normally would, e.g., just double click on one of the files:
Figure 3. Sample output for one of the prefixes from AS3701
If you requested JSON output instead of text, you may want to use jq or similar tools to “pretty print” or postprocess that output.
IV. Conclusion
We hope you continue to find the modified-to-use-stat.ripe.net sample Scala “Query-by-ASN” application useful!
If you have any questions or problems installing or using this sample application, feel free to drop the author a note.
Joe St Sauver Ph.D. is a Distinguished Scientist for Farsight Security, Inc.
