User Guides

Security Information Exchange (SIE) Technical Overview

Cybercriminals create and tear down their Internet infrastructures at rapid rates. As they refine and evolve their tactics, techniques, and procedures (TTPs), data must be collected and actionable intelligence shared in real-time so security teams can detect and respond to threats with equal speed.

Security professionals have access to an abundance of threat intelligence; however, a lot of the data is from the past, which is equivalent to looking in the rearview mirror while trying to drive forward. These teams and their platforms must be equipped with real-time insights into global Internet activity to understand the impact of threats and make actionable use of the intelligence using informed and relevant decisions.

The Security Information Exchange (SIE), from Farsight Security® Inc. (now part of DomainTools), is a scalable and adaptable real-time data streaming and information sharing platform. SIE collects and provides access to more than 200,000 observations per-second of raw data from its global sensor network. Farsight also applies unique and proprietary methods for improving usability of the data, directly sharing the refined intelligence with SIE customers and DNSDB®, one of the world’s largest passive DNS (pDNS) databases.

The diverse set of data available from SIE includes the following and is relevant and useful for practitioners in various technology roles:

Raw and processed passive DNS data
Darknet/darkspace telescope data
SPAM sources and URLs
Phishing URLs and associated targeted brands
Connection attempts from malware-infected systems (as seen by a sinkhole)
Network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices

Each unique set of data in SIE is known as a channel and the data acquired from a specific channel can be customized to meet the needs of each customer, enabling you to subscribe to and access only the channels needed to solve your problem. A channel in SIE may be the result from analyzing the data or a subset of data from other channels.

The data available from SIE channel subscription packages includes:

Raw Passive DNS: Real-time observations of DNS cache-miss traffic sent from DNS recursive resolvers on the Internet to authoritative name servers. The DNS information includes authoritative DNS data that various zone operators make available and responses from authoritative name servers to recursive resolver queries
Value-Added Passive DNS: Real-time de-duplicated, filtered, and verified passive DNS (pDNS) data observed on the Internet
Newly Observed Domains (NOD) and Newly Observed Hostnames (NOH): Real-time actionable insights for domains and hostnames, fully qualified domain names (FQDNs), when they are first successfully resolved on the Internet
Base Channels: A collection of threat intelligence channels that provide access to honeypot data (darknet and spam) and botnet (e.g., Conficker) sinkhole data. The data also includes threat intelligence for phishing campaigns and log data for network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices
Premium Channels: A range of premium security-related feeds including malware metadata, IOCs and other telemetry. Subscribers consume the intelligence as real-time event flows rather than traditional batch transfers – which are inherently delayed

The Security Information Exchange (SIE), from Farsight Security Inc., is designed with privacy in mind. The passive DNS (pDNS) sensors do not collect Personally Identifiable Information (PII) from client resolvers (also known as stub) by deliberately collecting between recursive resolvers and authoritative servers.

The data from SIE enables security professionals to accurately identify, map, and protect their networks from cybercriminal activity by providing global visibility. It provides immediate access to a real-time global sensor network without the need to develop or deploy your own data collection infrastructure.

Why Passive DNS (pDNS)?

DNS is a critical component of Internet communication and almost all Internet transactions begin with a DNS query and response.

Visiting a website? Your system uses DNS to resolve the IP address of the hostname for the website you are attempting to access
Sending an email? Email uses DNS to resolve the IP address of the mail exchange server your message should be delivered to

DNS serves as early warning and detection solution for phishing, spam, malicious and suspicious behaviors, and other attacks. DNS intelligence is considered the only source of “ground truth” information for the Internet.

Passive DNS (pDNS) begins with raw DNS traffic that is observed and collected by passive DNS sensors and contributed to Farsight’s Security Information Exchange (SIE) by pDNS sensor operators. Once the data is sent to SIE, the data then passes through a series of processing phases:

Deduplication: Channel 207, DNSDB Deduplicated Data
Verification: Channel 208, DNSDB Verified Data
Filtering: Channel 204, Processed DNS Data (which used by DNSDB)

The end result is the highest-quality and most comprehensive passive DNS database, DNSDB, of its kind-with more than 100 billion unique DNS resource records since 2010.

Farsight Security’s mission is to make the Internet a safer place. We provide security solutions that empower customers with meaningful and relevant intelligence. This information provides customers with insights about the network configuration of a threat and the surrounding network on the Internet for improving the value and impact of threat intelligence and research.

The Passive DNS Processing “Waterfall” Model

Data is processed by Security Information Exchange (SIE) in what is called a waterfall model.

The following diagram can help inform and guide you in understanding the data that is available from the various SIE DNS channels. Farsight’s Solution Architect’s (SAs) are happy to discuss criteria for selecting the appropriate SIE Channels with customers.

The deduplication phase performs data reduction and exports the unique DNS records with counts for the number of times each unique DNS answer was observed in the data.

However, some types of DNS data are also filtered at this state, such as DNS messages that have a bad checksum value or data that has been delayed for more than an hour. These records are discarded.

Rogue, malicious, or misconfigured name servers may respond with misleading resource record information for a domain or domains. The verification phase ensures that only bailiwick-appropriate DNS data is passed on to Channel 208, DNSDB Verified Data. DNS data that fails bailiwick verification is discarded.

The next phase is filtering and the final phase in the waterfall model processing. In this phase, various categories of DNS data are filtered, which may including the following:

Dynamic addresses, which is rarely of interest to passive DNS (pDNS) users
encrypted DNS tunneling traffic
DNS-as-general-purpose-database traffic

DNS records that are not filtered are sent to Channel 204, Processed DNS Data, which is the channel also used by DNSDB.

DNS Terminology: RRname (RRset), Rdata, RRtype and Bailiwick

There are DNS terms used in this document that my be unfamiliar to the reader. Definitions and links to additional information for these terms follows:

RRname: Also known as the question or “RRset“, refers to the left-side of a resource record in a DNS response and contains the owner name, TTL, CLASS, TYPE and RDATA. An RRset contains one or more resource records with the same owner name, class, type, rdata, usually the same TTL, and observed at the same time
Rdata: Also known as the answer or “record data“, refers to the right-side of a resource record in a DNS response
RRtype: This field in the DNS resource record specifies the type of data that is being returned. Common RRtype values include A (host address), AAAA (IPv6 host address), CNAME (Canonical name), and NS (Name server)
Bailiwick: DNS data is considered “in bailiwick” if the resource record being returned is the response from a name server that is known to be responsible for answering with authoritative information about that domain

More information:

In the example DNS resource record that follows, RRname (left-side) refers to www.farsightsecurity.com and Rdata (right-side) refers to 66.160.140.81 or the IP address. Rdata can also refer to a Fully Qualified Domain Name (FQDN) such as info.farsightsecurity.com. See RRset and Rdata Demystified for additional information.

Example DNS Resource Record: RRname, RRclass, RRtype, and Rdata

RRname/RRset (Left-Side)	RRclass	RRtype	Rdata (Right-Side)
www.farsightsecurity.com	IN	A	66.160.140.81

SIE Access Methods

Data from SIE can be accessed and acquired using the following methods:

Direct Connect: Connect a system to the SIE network. This 1.) requires a server to be installed in a data center where Farsight has a point of presence, and 2.) then ordering a network cross connect between your server and the SIE network. Customers can optionally, and prefer to, lease a blade server from Farsight
SIE Remote Access (SRA): Remotely connect to the SIE network using an encrypted tunnel from your workstation or a server in your local data center
SIE Batch: Provides on-demand access for downloading data from SIE channels using a RESTful API or web-based interface. You select the channel and duration of time you are interested in, and then download the data for analysis. The duration of available data is dependent on the channel, but is typically the most recent 12-18 hours

For additional information about SIE access methods, please see the SIE Technical Overview document.

Direct Connect

SIE Direct Connect allows a customer to physically connect a server to the Farsight SIE network for maximum data throughput. This can be done in one of two ways:

Blade Server: Pre-configured blade servers co-located in one of Farsight’s data centers that can be leased by customers for direct access to SIE channels
Customer Server: Customer (owned, managed, and operated) servers that can be installed in one of Farsight’s data centers and physically connected to the SIE network with a network cross-connect

If a blade server is leased from Farsight, it will be pre-installed with the essential software components needed to acquire, process, compress, buffer, and transfer data from SIE channels to the customer’s data center for additional analysis, enrichment, and storage.

If a customer uses their own server, an order can be submitted for a cross-connect to the SIE switches hosted at select Equinix data centers (Ashburn DC3 and Palo Alto SV8). An FSI account manager can help guide cross-connect provisioning details, hosting, or colocation options.

For additional information about SIE connection methods, please please refer to the SIE Technical Overview guide. A Farsight’s sales representatives is happy to share a copy of this document with you. This will help inform and guide you in understanding which connection method will work best for you.

SIE Remote Access (SRA)

SIE Remote Access (SRA) enables a customer to remotely connect to the Security Information Exchange (SIE) from anywhere on the Internet. SRA provides access to SIE channel data on customer’s local servers, allowing their analysis and processing systems to be located in their own data centers rather than physically co-located at a Farsight’s data center.

Due to the technical limitations of transporting high bitrate SIE channels across the Internet, the SRA access method is not available for all SIE channels. Please reference the SIE Channel Guide for channels that can be accessed using SRA.

SRA uses the Advanced Exchange Access (AXA) transport protocol which enables SRA sessions to perform the following:

Select which SIE channel or channels to monitor and acquire data from
Define user-specified search or filtering criteria to match IP or DNS traffic
Control rate-limits and other AXA parameters

The streaming search and filtering capabilities of AXA enables SRA to access and acquire meaningful and relevant data from SIE while avoiding the costs of transporting enormous volumes of data across the Internet.

Note: For high volume channels accessed using SRA, it is expected that customer’s will specify a search or filter for IP addresses and DNS domain names or hostnames of interest. The SRA service will only collect and send data matching the specified criteria across the Internet to the customer.

SIE Batch

SIE Batch provides on-demand access for downloading data from SIE channels using a RESTful API or web-based interface. You select the channel and duration of time you are interested in, and then download the data for analysis. The duration of available data is dependent on the channel, but is typically the most recent 12-18 hours. SIE Batch allows you to acquire data from SIE channel using two (2) methods:

API: Allows you to write tools to programmatically download data from SIE channels for analysis
Interactively: Web-based interface to the API that enables you to select and download SIE channel data on-demand

Advanced Exchange Access Middleware Daemon (AXAMD)

Farsight also provides a RESTful middleware layer in front of its AXA service. This service is called the AXA Middleware Daemon (AXAMD) and provides a RESTful capability that adds a streaming HTTP interface on top of the AXA toolkit. This enables web-application developers to interface with SIE using SRA. Farsight also published a command line tool and Python extension library called axamd_client. This toolkit is licensed under the Apache 2.0 license.

The Advanced Exchange Access (AXA) toolkit contains tools and a C library to bring Farsight’s real-time data and services directly from the Farsight Security Information Exchange (SIE) to the customers network.

Advanced Exchange Access Middleware Daemon (AXAMD) is a suite of tools and library code to bring Farsight’s real-time data and services directly from the Farsight Security Information Exchange (SIE) to the customers network.

Due to the technical limitations of transporting high bitrate SIE channels across the Internet, the AXAMD access method is not available for all SIE channels.

Advanced Exchange Access Toolkit (AXA)

Farsight’s Advanced Exchange Access Toolkit (AXA) enables customers to remotely and securely connect to the SRA (SIE Remote Access) service. The SRA service provides access to channels available from Farsight’s Security Information Exchange (SIE). AXA is a Farsight developed binary protocol used to transport real-time data available from SIE.

AXA uses a streaming API encrypted by TLS for transporting SIE data over the Internet. The AXA protocol uses two (2) streams that transport messages between a customers client, such as sratool, and the SRA service. There is one (1) stream in each direction using a single TCP connection.

Farsight also provides a RESTful middleware layer in front of its AXA service. This service is called the AXA Middleware Daemon (AXAMD) and provides a RESTful capability that adds a streaming HTTP interface on top of the AXA toolkit. This enables web-application developers to interface with SIE using SRA. Farsight also published a command line tool and Python extension library called axamd_client. This toolkit is licensed under the Apache 2.0 license.

NMSG

To acquire, prepare, and transport SIE data, Farsight created an adaptable container wire and file format for storing and transmitting blobs of data called Network Message (NMSG). As its core, NMSG leverages Google Protocol Buffers Version 2 for binary encoding using pre-defined schemas, or in a native packetized format like PCAP.

Other data formats, like JSON or XML, can also be encapsulated in NMSG for consistent transport across Farsight’s Security Information Exchange (SIE) infrastructure and acquired and analyzed by receiving systems.

The adaptable NMSG container format allows for consistent or variable message types. NMSG container data may be streamed to a file or transmitted as UDP datagrams. NMSG containers can contain multiple NMSG messages or a fragment of a message too large to fit in a single container. The data in an NMSG container can also be compressed. Additional capabilities include sequencing and rate-limiting.

More information is available in the Farsight’s Network Message, Volume 1: Introduction to NMSG blog article.

Additional Information

Security Information Exchange (SIE)
- SIE User Guide
- SIE Channel Guide
Advanced Exchange Access (AXA)
Network Message (NMSG)
Newly Observed Domains (NOD) & Newly Observed Hostnames (NOH)
- Newly Observed Domains (NOD)
- Newly Observed Hostnames (NOH)
Darknet
- Spotting A DNS Denial of Service Reflection Attack in SIE Darkspace Telescope Data
- Accessing Darknet Telescope Data via SIE Remote Access (SRA)

Security Information Exchange (SIE) Technical Overview

Share this entry