SIE Batch Technical Overview
The Security Information Exchange (SIE), from Farsight Security® Inc. (now a part of DomainTools), is a highly scalable security information sharing platform. It can be thought of as “radar for the Internet”, a way for you to study what’s happening online. Farsight collects and redistributes more than 200,000 new raw observations per second from its global network of sensors. Farsight also applies unique proprietary methods to improve the usability of that data, sharing refined intelligence with SIE customers directly and via DNSDB, one of the world’s largest passive DNS databases.
SIE distributes a variety of types of data of use for the security
professional, including:
- Raw and processed passive DNS data
- Darknet/darkspace telescope data
- SPAM sources and URLs
- Phishing URLs
- Connections from malware-infected systems (as seen by a sinkhole)
- Intrusion detection system (IDS) and firewall connection block data
SIE Batch is a new delivery method that gives you access to a RESTful API that can be used
to download data as needed. It also has a web-based interface that can be used to define
your data sets and download them. With SIE Batch you can select the data sets and time
periods of interest to you, download that data and have it available for your analysis.
SIE Batch allows you to access data two ways:
- Via the SIE Batch API: The API allows you to write programs to pull down data for
for processing automatically. - Interactively: There is a web-based interface that acts as a front end to the API and
allows you to select and download sets of data on demand.
SIE Batch gives you access to the most recent data distributed via the SIE system. How
much data is available depends on the channel you’re pulling data from, but is typically
the most recent 12-18 hours.
Accessing SIE Data Interactively via SIE Batch
The SIE Batch system requires a subscription to the SIE data. When you set up the subscription you will receive an API key which will give you access to the system. If you don’t have an active subscription, please contact the DomainTools sales team.
Once you are logged in you will see the SIE Batch dashboard. SIE data is returned in one of two formats: Newline Delimited JSON (ND-JSON) and NMSG. ND-JSON formatted files have a suffix of .ndjson, while NMSG formatted files have a suffix of .nmsg.
Once you have the files downloaded, you can hand them off to whatever program you have created to evaluate and process the data in them.
Newline Delimited JSON (ND-JSON) formatted files
ND-JSON files are formatted text files. The specific fields within the data will vary by channel, but it will look something like this sample, which is from Channel 213, Newly Observed Domains:
{
"time": "2020-01-13 17:53:00.097326040",
"vname": "SIE",
"mname": "newdomain",
"source": "a1ba02cf",
"message": {
"domain": "clienttons.com.",
"time_seen": "2020-01-13 16:16:04",
"bailiwick": "ipv4-only.cname.clienttons.com.",
"rrname": "jdkyqftipq6rwxq4s7ca-pw7etn-d8f0af301.ipv4-only.cname.clienttons.com.",
"rrclass": "IN",
"rrtype": "CNAME",
"rdata": [
"a248.b.akamai.net."
],
"keys": [],
"new_rr": []
}
}ND-JSON files can be viewed directly or used with any tool that supports the ND-JSON format.
NMSG Formatted files
NMSG files are a binary format, so they can’t be used directly. Farsight has released tools to read NMSG formatted content. The NMSG tool can be found on Github at https://github.com/farsightsec/nmsg. If you are using Debian, there are packages that can be installed via apt-get. See https://www.farsightsecurity.com/technical/SIE-user-guide/sie-debian/ for instructions.
To look at NMSG data, you run nmsgtool, which will format an NMSG file as readable test.
If you were to view a file downloaded from Channel 221 (NSDomains) via the command
nmsgtool -r , you will see something like this:
[70] [2020-01-13 17:46:47.996805233] [2:6 SIE dnsnx] [a1ba02cf] [] []
qname: 246.25.155.49.in-addr.arpa.
qclass: IN (1)
qtype: PTR (12)
response_ip: 194.146.106.106
soa_rrname: 49.in-addr.arpa.Conclusion
SIE Batch is an additional access method to Farsight’s SIE data, making SIE data available to threat intelligence analysts, Firewall policy systems, Malware professionals and other security teams without the server overhead and processing complexity of a real-time feed. If you do need realtime or even near-realtime feeds, then as mentioned above, you will likely achieve better results from using SIE Remote Access (SRA).
The data channels available through SIE allow these groups access to information to help identify and manage various threats on the Internet, including spam sources and URLs, Phishing attacks and malware. SIE data can help inform Intrusion detection system (IDS) and firewalls, allow or deny connections to email and data servers, and many other activities that happen within an organization on a daily basis.
If you are interested in learning more about SIE and SIE Batch, please contact the DomainTools sales team.
Additional Information
- SIE Batch REST API Reference
- SIE User Guide
- SIE Channel Guide
- NMSG
- JQ (Json editor/formatter)
