Security Information Exchange (SIE) DNS Errors
Channel 220, the SIE DNS Errors channel is a source of DNS intelligence for DNS queries that result in an error being returned. DNS Errors is a real-time non-deduplicated channel of DNS responses that returned a non-zero Response Code (RCODE). This includes NXDomain, ServFail, Refused, FormErr, NotImp, and other RCODEs. A lot of interesting information can be discovered and learned about DNS by investigating these errors.
About Security Information Exchange (SIE)
The Security Information Exchange (SIE), from Farsight Security® Inc. (now part of DomainTools), is a scalable and adaptable real-time data streaming and information sharing platform. SIE collects and provides access to more than 200,000 observations per-second of raw data from its global sensor network. Farsight also applies unique and proprietary methods for improving usability of the data, directly sharing the refined intelligence with SIE customers and DNSDB®, one of the world’s largest passive DNS (pDNS) databases.
The diverse set of data available from SIE includes the following and is relevant and useful for practitioners in various technology roles:
- Raw and processed passive DNS data
- Darknet/darkspace telescope data
- SPAM sources and URLs
- Phishing URLs and associated targeted brands
- Connection attempts from malware-infected systems (as seen by a sinkhole)
- Network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices
Each unique set of data in SIE is known as a channel and the data acquired from a specific channel can be customized to meet the needs of each customer, enabling you to subscribe to and access only the channels needed to solve your problem. A channel in SIE may be the result from analyzing the data or a subset of data from other channels.
Why Passive DNS (pDNS)?
DNS is a critical component of Internet communication and almost all Internet transactions begin with a DNS query and response.
- Visiting a website? Your system uses DNS to resolve the IP address of the hostname for the website you are attempting to access
- Sending an email? Email uses DNS to resolve the IP address of the mail exchange server your message should be delivered to
DNS serves as early warning and detection solution for phishing, spam, malicious and suspicious behaviors, and other attacks. DNS intelligence is considered the only source of “ground truth” information for the Internet.
Passive DNS (pDNS) begins with raw DNS traffic that is observed and collected by passive DNS sensors and contributed to Farsight’s Security Information Exchange (SIE) by pDNS sensor operators.
Farsight Security’s mission is to make the Internet a safer place. We provide security solutions that empower customers with meaningful and relevant intelligence. This information provides customers with insights about the network configuration of a threat and the surrounding network on the Internet for improving the value and impact of threat intelligence and research.
The Security Information Exchange (SIE), from Farsight Security Inc., is designed with privacy in mind. The passive DNS (pDNS) sensors do not collect Personally Identifiable Information (PII) from client resolvers (also known as stub) by deliberately collecting between recursive resolvers and authoritative servers.
The data from SIE enables security professionals to accurately identify, map, and protect their networks from cybercriminal activity by providing global visibility. It provides immediate access to a real-time global sensor network without the need to develop or deploy your own data collection infrastructure.
About SIE DNS Errors
Channel 220 provides insights for DNS queries that return an error. Billions of DNS queries occur every day and most return usable answers. However, in some cases DNS returns an Response Code (RCODE) error because the domain you are interested in does not exist “NXDomain“, the name server was unable to process a query due to an issue with the name server “ServFail“, or name server refuses to perform the specified operation for policy reason “Refused“. A lot of interesting information can be discovered and learned about DNS by investigating these errors.
RCODEs include but are not limited to NXDomain (3), ServFail (2), Refused (5), FormErr (1), NotImp (4), and others. Look here for a current list of DNS RCODEs.
Use Cases for SIE DNS Errors
Customers can discover and learn a lot of interesting information about DNS by investigating these errors. By analyzing this DNS intelligence, customers can solve operational, security, risk, and brand protection related issues.
- DNS Operational Monitoring: Customers can monitor their domains on this channel and be immediately notified if DNS for your domains or hostnames fail anywhere on the globe
- Brand Protection: Your company has brands and trademarks that it needs to protect from infringement and counterfeit. Customers can monitor Channel 220 DNS Errors and be notified when someone starts investigating domains that would infringe on your intellectual property
- Domain Monetization: Some companies have created businesses that focus on mistyped domain names. They register commonly misspelled domains, create websites, and then serve ads when users accidentally visit the website. Farsight does not license data to companies that perform this type of nefarious behavior. However, DNS Errors can inform and help customers identify commonly misspelled domains and monitor them to look for these predatory practices
- Detection of Domain Generation Algorithms (DGAs): Criminals that build Botnets often use Domain Generation Algorithms (DGAs) to spread their command and control (C&C) servers across many domains. The purpose is to ensure that if a C&C server is taken down by authorities, the infected systems can fall back and reconnect to an alternative C&C server. The DNS Errors channel enables analysts and researchers to monitor failed DNS requests to help them identify these hidden C&C networks
Detection of Domain Generation Algorithms (DGAs)
One significant use-case for channel 220 is identifying users infected with malware. Infected computers will attempt to contact command and control (C&C or C2) servers to receive instructions or commands on what it should do.
Criminals that create malware may use a Domain Generation Algorithm (DGA) to generate a large number of fallback domain names. This effort is an attempt to ensure the criminal does not lose control of the botnet when the primary C&C server is taken down by authorities. If the domain name of primary C&C server can not be resolved, the malware may attempt to resolve a fallback domain name created by the DGA and then connect to a secondary C&C server.
This behavior can generate an excessive amount of DNS resolution failures, which can be used to identify infected systems attempting to contact C&C servers.
DGAs created by malware authors are designed to prevent reverse-engineering. This makes is difficult for analysts and security practitioners to discover and identify C&C servers from DNS responses returning a non-zero Response Code (RCODE).
Additionally, malware has started to generate and test numerous domains to make it even more challenging to reverse engineer the DGA and track C&C servers. While that makes it more difficult to identify the C&C servers, the excessive amount of DNS erros makes it easier to identify the infected computers so they can be remediated.
Read this for more on Domain Generation Algorithms
Channel Information for SIE Channel 220
SIE distributes a variety of types of data of use to the security professional, including:
- Data on domain creation and updates
- Raw and processed passive DNS data
- Darknet/darkspace telescope data
- SPAM sources and URLs
- Phishing URLs
- Connections from malware-infected systems (as seen by a sinkhole)
- Intrusion detection system (IDS) and firewall connection block data
SIE transports these different data sets as feeds, known as channels. You can subscribe to only the channels you need to solve your security challenges.
SIE DNS Errors Use Cases
Hundreds of billions of Domain Name System (DNS) queries are made every day. Most DNS queries return usable answers, but in some cases, DNS returns an error because the domain name you’re interested in does not exist.
You can learn a lot by investigating these errors. By studying this data customers can solve a number of operational and security issues facing online operations today:
- DNS Operational Monitoring: You can monitor your own domain names on these channels and be immediately notified if for some reason the DNS for your sites fails anywhere on the globe
- Brand Protection: Your company has brands and trademarks that it needs to protect from infringement. The DNS Errors channel traffic is an easy way to monitor and be alerted when someone starts investigating domains that would infringe on your intellectual property
- Domain Monetization: Some companies have created entire businesses that focus on mistyped domain names — they register common misspellings, create websites and then serve ads to those users who accidentally visit them. Farsight doesn’t license data to companies that do this, but it can help you identify the common mis-spellings of your own domain names and monitor them to look for these predatory practices
- Detection of Domain Generation Algorithms (DGAs): Criminals that build Botnets often use Domain Generation Algorithms (DGAs) to spread their command and control channels across many domains so that if one control node is identified and disabled, the infected systems can fall back and reconnect via an alternative control node. The DNS Errors channel allows analysts and researchers to monitor failed DNS requests to help them identify these hidden control networks
Use Case: Detection of Domain Generation Algorithms (DGAs)
A common use of DNS Errors is identifying users infected with malware. These infected computers will attempt to contact the command and control server to get orders on what they should do.
Malware systems use a Domain Generation Algorithm (DGA) to ensure they don’t lose control of the botnet when the primary control server is taken down by authorities. If the command server can’t be found, the malware uses the DGA to generate fallback domain names that the C&C server might be using, and it will attempt to contact it there.
This will generate large numbers of DNS failures in the lookup attempts, which can help identify both the C&C servers and the infected systems attempting to contact it.
The specific DGA algorithms are secret so that analysts can’t figure out where the servers might be from the error traffic being generated. Additionally, malware has started generating and testing many domains to make it harder to reverse engineer the DGA to track the servers.
While that makes it harder to locate the command servers, this high level of DNS error traffic makes it easier to identify the infected botnet systems so they can be corrected.
Data Format for the SIE DNS Errors Channel
The DNS Errors channel data uses the NMSG dnsqr data format.
DNS Query and Response resource record schema that observes and collects data returned
from a query. The data available from this channel contains NMSG base:dnsqr type
messages that include the following fields:
The NMSG header includes the following fields:
| KEY | VALUE |
| UDP_INVALID | Response data does not match the query data. |
| UDP_QUERY_RESPONSE | 9-tuple (query_ip, response_ip, proto, query_port response_port, id, qname, qtype, qclass) all match in the DNS query and response messages. |
| UDP_UNANSWERED_QUERY | DNS queries that did not receive a response within the state table window. |
| UDP_UNSOLICITED_RESPONSE | DNS response received, but no corresponding DNS query exists in the state table. |
| TCP | Response to DNS query message was sent using TCP. |
| ICMP | Response to DNS query message was sent using ICMP. |
See Domain Name System (DNS) Parameters for information about Resource Record (RR) TYPEs qtype and DNS Response Codes (RCODEs) rcode.
For information on the possible values for qtype and rcode see Domain Name System (DNS) Parameters, linked to at the bottom of this document.
query_packet, response_packet, query, response, and dns are base64 encoded binary data extracted from the query and response data by the sensor. They are equivalent to the data seen by a package capture tool like pcap. Discussion of decoding and viewing this data is beyond the scope of this document.
Using SIE DNS Errors Data
DNS Errors data is delivered in NMSG format, which is a compact binary data form. To process the data you need to convert it into a usable form. Farsight has released tools as open source that do this. The NMSG package and the program nmsgtool will read the nmsg format and display the data either text or JSON format for viewing or further processing. It uses a library from the sie-nmsg package to support the data found on the DNS Errors channel.
If you are using the Debian Linux distribution, you can get instructions for installing this software through apt-get using the instructions found in Security Information Exchange (SIE) on Debian. For other Linux-style distributions these instructions might work, or you may need to download and compile the software directly using the instructions in the Git repositories. Links to all of these items are below.
To see the DNS Errors data, you can decode it via nmsgtool. This example assumes
downloading data via the SIE Batch delivery system. The -r option defines the location
of the data, the -c 1 asks for a single record, and the -J - outputs the record in
JSON Lines (Newline Delimited JSON) format:
| Channel Name | DNS Errors |
| Channel Number | 220 |
| Description | Domain names where authoritative servers answered with an error code. |
| Schema | base:dnsqr |
To see current channel traffic volumes and service options for accessing it, please see the Security Information Exchange (SIE) Channel Guide.
Example Message from SIE Channel 220
Data acquired from Channel 220 DNS Errors is returned in NMSG format. NMSG is an adaptable container format that allows for consistent or variable message types.
The nmsgtool program is a tool for acquiring a variety of different inputs,
like data streams from the network, capturing data from network interfaces,
reading data from files, or even standard input and making NMSG payloads
available to one or more outputs. The nmsgtool program can acquire data from
SIE Channel 220 and convert it to a ND-JSON (newline-delimited JSON) text format
for display or additional processing and analysis. nmsgtool is a program
written by Farsight and released as open source.
See the following pages for instructions on how to install software packages for a specific distribution.
- Security Information Exchange (SIE) on Debian
- Security Information Exchange (SIE) on CentOS 6 / RHEL 6
- Security Information Exchange (SIE) on FreeBSD
After data for Channel 220 has been acquired, written, and saved to a file, you
need to decode it to ND-JSON using nmsgtool. The [-r ch220_errors.nmsg]
option tells nmsgtool to read binary NMSG data from a file, [-c 1] limits
the output to single NMSG payload, and [-J -] displays the record in ND-JSON
format to stdout, which is typically the screen.
$ nmsgtool -r ch220_errors.nmsg -c 1 -J -
{"time":"2020-04-14 16:55:36.223682000","vname":"base","mname":"dnsqr",
"source":"a1ba02cf","message":{"type":"UDP_QUERY_RESPONSE",
"query_ip":"10.10.10.10","response_ip":"10.10.10.10","proto":"UDP",
"query_port":43654,"response_port":53,"id":3569,"qname":"example.com.",
"qclass":"IN","qtype":"A","rcode":"SERVFAIL",
"query_packet":["RQAAS5TxQABAEX0yTVg4Q5awDDOqhgA1ADcoxw3xAAAAAQAAMEbAAEAAQAAKQaQAACAAAAA"],
"query_time_sec":[1586883336],"query_time_nsec":[223682000],
"response_packet":["RWAAS4jrAAD5EQ/YlrAMM01YOEMANaqGQA2NwcwNrMTICZmwAQAAKRAAAACAAAAA"],
"response_time_sec":[1586883336],"response_time_nsec":[397941000],
"delay":0.17425899999999999723,"udp_checksum":"CORRECT",
"query":"DfEAAAABAAAAAAABA2VsYwRvY3BzA2sxMgJmpBpAAAIAAAAA=",
"response":"DfGAAgABAAAAAAABA2VsYwRvYgJmbAJ1cwAAAQABAAApEAAAAIAAAAA=",
"dns":"DfGAAgABAAAAAAABA2VsxMgJmbAJ1cwAAAQABAAApEAAAAIAAAAA="}}
If you want to display a pretty-printed output of ND-JSON formatted records, we
recommend using jq (a lightweight and flexible command-line JSON processor
). The open source software package is
available on Debian and can be installed using $ sudo apt-get install jq. The
output from nmsgtool in JSON format [-J -] can be piped to jq using the
following:
$ nmsgtool -r ch220_errors.nmsg -c 1 -J - | jq -r '.'
{
"time": "2020-04-14 16:55:36.223682000",
"vname": "base",
"mname": "dnsqr",
"source": "a1ba02cf",
"message": {
"type": "UDP_QUERY_RESPONSE",
"query_ip": "10.10.10.10",
"response_ip": "10.10.10.10",
"proto": "UDP",
"query_port": 43654,
"response_port": 53,
"id": 3569,
"qname": "example.com.",
"qclass": "IN",
"qtype": "A",
"rcode": "SERVFAIL",
"query_packet": [
"RQAAS5TxQABAEX0yTVg4Q5awDDOqhgA1ADcoxw3xAAAAAQAAMEbAAEAAQAAKQaQAACAAAAA"
],
"query_time_sec": [
1586883336
],
"query_time_nsec": [
223682000
],
"response_packet": [
"RWAAS4jrAAD5EQ/YlrAMM01YOEMANaqGQA2NwcwNrMTICZmwAQAAKRAAAACAAAAA"
],
"response_time_sec": [
1586883336
],
"response_time_nsec": [
397941000
],
"delay": 0.174259,
"udp_checksum": "CORRECT",
"query": "DfEAAAABAAAAAAABA2VsYwRvY3BzA2sxMgJmpBpAAAIAAAAA=",
"response": "DfGAAgABAAAAAAABA2VsYwRvYgJmbAJ1cwAAAQABAAApEAAAAIAAAAA=",
"dns": "DfGAAgABAAAAAAABA2VsxMgJmbAJ1cwAAAQABAAApEAAAAIAAAAA="
}
}
SIE Delivery Options
SIE data can be accessed through a variety of access methods depending on your needs with a subscription to Farsight’s systems.
SIE Batch
SIE Batch allows you to download selections of recent SIE data on demand, either manually or via a program and the SIE Batch API. It provides asynchronous access to SIE data for those users that don’t want or need to ingest and process a real-time feed of the SIE data. You can select the channels, datasets and time periods of interest to you and collect them either via the API or web-based dashboard.
Due to the inability to distribute the high volume channels reliably in this way, not all SIE channels are available via SIE Batch.
Data from SIE can be accessed and acquired using the following methods:
- Direct Connect: Connect a system to the SIE network. This 1.) requires a server to be installed in a data center where Farsight has a point of presence, and 2.) then ordering a network cross connect between your server and the SIE network. Customers can optionally, and prefer to, lease a blade server from Farsight
For additional information about SIE access methods, please see the SIE Technical Overview document.
Direct Connect
SIE Direct Connect allows a customer to physically connect a server to the Farsight SIE network for maximum data throughput. This can be done in one of two ways:
- Blade Server: Pre-configured blade servers co-located in one of Farsight’s data centers that can be leased by customers for direct access to SIE channels
- Customer Server: Customer (owned, managed, and operated) servers that can be installed in one of Farsight’s data centers and physically connected to the SIE network with a network cross-connect
If a blade server is leased from Farsight, it will be pre-installed with the essential software components needed to acquire, process, compress, buffer, and transfer data from SIE channels to the customer’s data center for additional analysis, enrichment, and storage.
If a customer uses their own server, an order can be submitted for a cross-connect to the SIE switches hosted at select Equinix data centers (Ashburn DC3 and Palo Alto SV8). An FSI account manager can help guide cross-connect provisioning details, hosting, or colocation options.
For additional information about SIE connection methods, please see the SIE Technical Overview document. A Farsight’s sales representatives is happy to share a copy of this document with you. This will help inform and guide you in understanding which connection method will work best for you.
Additional Information
Links to Information in this Document
- Domain Name System (DNS) Parameters
- Domain Name System (DNS) Parameters: DNS Response Codes (RCODEs)
- Farsight’s NMSG GitHub Repository
- Farsight’s SIE-NMSG GitHub Repository
SIE:newdomainDNS Query and Response RR schemajqa lightweight and flexible command-line JSON processor- Description of JSON Lines text format (aka ND-JSON)
DNS Errors Links
- What (Besides NXDOMAINs) Do We See on Farsight Security’s DNS Errors Channel?
- Finding Web Proxy Auto Discovery Protocol (WPAD)-related Security Exposures Using Farsight Security’s NXDOMAINs Channel
- DNS is NOT Boring! Using DNS to Expose and Thwart Attacks* from 29th Annual FIRST Conference, 2017
