Newly Observed Domains (NOD) Frequently Asked Questions (FAQ)
S. I) Product Definition Questions
Q. I-1) What’s the difference between Newly Observed Domains (NOD) and Newly Observed Hostnames (NOH)?
If we consider the hypothetical name www.example.com:
- .com would be the top level domain (“TLD”) or “public suffix”
- example.com would be the domain, registered by a registrant directly above a TLD or public suffix
- www.example.com would be the hostname (or Fully Qualified Domain Name, “FQDN”) as might be used by a workstation, web server, mail server, networked printer, or other system.
Newly Observed Domains only lists newly seen domains, while Newly Observed Hostnames tracks the creation of individual hostnames on a hostname-by-hostname basis.
Traditional generic top level domains, or gTLDs, include com, net, org, edu, gov, and mil. That original set of gTLDs has now expanded over time to well over 1,000 different gTLDs.
In addition to gTLDs, country code TLDs (“ccTLDs”), based on ISO-assigned country codes, also exist. For example, .br is for Brazil, .ca is Canada, .ch is for Switzerland (“Confoederatio Helvetica”), .de is for Germany (“Deutschland”), .fr is France, .mx is Mexico, etc. See Wikipedia’s List of Internet Top Level Domains for the full list.
Some ccTLDs structure their effective top level domain names differently. UK companies registering domains under the .uk ccTLD do so under .co.uk, rather than directly under the .uk TLD.
There are also other examples of special case “effective TLDs,” including services that offer “dynamic DNS service.” Dynamic DNS service in this case means helping individuals to easily assign (and subsequently update) a domain name to dynamic addresses received via DHCP. Examples of some such services can be seen at http://dnslookup.me/dynamic-dns/ .
Farsight use the Public Suffix List to help us keep those sort of special “cut points” straight.
Q. I-2) What does “new” mean in Newly Observed Domains or Newly Observed Hostnames?
This is a domain or hostname seen in passive DNS that hasn’t previously been seen by a Farsight sensor node since June 2010, and which hasn’t already been seen in a zone file obtained under the Zone File Access programs.
Q. I-3) How is NOD better than just lists of new domains from Zone File Access programs?
Zone files are typically provided for download via the Zone File Access program just once a day. This can result in a big visibility gap: you’ll see some intensively-abused domains created, deployed, abused and then abandoned during the few short hours between the time they’re created and the time the zone file that first mentions them becomes available. Zone files are simply too “batch oriented” for a real-time world.
By way of contrast:
- Newly Observed Domains lists new domains virtually as soon as they are seen by one of our sensor nodes
- Newly Observed Domains includes domains from TLDs for which no zone file access program exists (such as many ccTLD zones)
- Newly Observed Domains also includes new domains created under “effective top level domains,” as defined by the Public Suffix List.
Q. I-4) In what formats can I get NOD? And what’s the difference between those formats?
There are four ways to access NOD:
- NOD RPZ
- NOD rbldnsd
- As a real-time channel at the Security Information Exchange, and
- By directly querying NOD via the public DNS.
NOD RPZ is meant to be used by a recursive resolver that has support for DNS Response Policy Zones. DNS Response Policy Zones allow nameservers to be used as “DNS firewalls,” intentionally blocking network access to select content based on blocklist-defined criteria (perhaps the blocked domain name is known to be associated with spam, phishing, malware, or other online abuse). In the case of NOD RPZ, NOD RPZ is used to automatically and briefly block access to domain names that are brand new. Why? A simple reality: legitimate domains don’t need users to be able to access that domain immediately, just after it has been created, but attackers rely upon being able to create a new domain and use it immediately, before domain reputation and security firms can discover, assess, and block that domain.
NOD rbldnsd is similar to NOD RPZ, except it is provided in a format that’s easily processed by the rbldnsd block list nameserver software as is often used to locally mirror blocklists for the purpose protecting mail servers from spam, phishing and other unwanted email.
If you may be confused about which of the two options you want, remember that NOD RPZ can protect all applications from accessing brand new domains, but requires that you run an RPZ-capable nameserver (such as current versions of BIND, Blue Cat DNS, or InfoBlox’s DNS Firewall). NOD rbldnsd integrates easily with existing email spam filtering at many sites, but requires that you run a local copy of rbldnsd.
NOD (and NOH) as SIE Channels. We also make NOD and NOH available as channels via the Farsight Security Security Information Exchange (SIE). Channel 212 is real-time streaming NOD data, and Channel 213 is real-time streaming NOH data. These channels allows researchers and those who’d like to directly work with raw NOD or NOH data to do so. An example of the presentation format of that data:
[101] [2016-04-29 18:17:54.005783081] [2:5 SIE newdomain] [a1ba02cf] [] []
domain: svetlanovskiy.accountant.
time_seen: 2016-04-29 18:16:55
bailiwick: svetlanovskiy.accountant.
rrname: svetlanovskiy.accountant.
rrclass: IN (1)
rrtype: A (1)
rdata: 162.13.208.85
NOD as a publicly queryable DNS zone. Finally, NOD is also available as a publicly queryable DNS zone. This publicly queryable zone is meant to be used for testing or exploratory use by a low-volume mail server, limiting any single IP address querying the zone to no more than 10 queries per second. While the publicly queryable zone would normally be queried from a program, you can also make test queries interactively:
$ dig +short test.dns-nod.net.v1.bl.dns-nod.net
127.0.0.2
$ dig +short test.dns-nod.net.v1.bl.dns-nod.net txt
"test record"
To show an actual example from April 29th, 2016:
$ dig +short svetlanovskiy.accountant.v1.bl.dns-nod.net
127.0.0.2
$ dig +short svetlanovskiy.accountant.v1.bl.dns-nod.net txt
"first_seen=1461953815"
$ date -r 1461953815
Fri Apr 29 11:16:55 PDT 2016
The coded 127.0.0.x values returned are:
| Time Frame | Response |
|---|---|
| 0-5 minute | 127.0.0.2 |
| 5-10 minutes | 127.0.0.3 |
| 10-30 minutes | 127.0.0.4 |
| 30-60 minutes | 127.0.0.5 |
| 1-3 hours | 127.0.0.6 |
| 3-12 hours | 127.0.0.7 |
| 12-24 hours | 127.0.0.8 |
NOH CSV is meant for those who prefer to rsync and work with a copy of Newly Observed Hostnames via a spreadsheet product such as Excel. The most recent CSV file is continually updated, with a new CSV file opened hourly and the past 24 hours worth of hourly files is also available.
Q. I-5) “I noticed that NOD RPZ is often delivered via zone transfer (AXFR/IXFR) while NOD rbldnsd is often transferred via rsync transfer mechanisms. Why?”
NOD RPZ is typically delivered to nameservers running BIND. Because BIND is able to easily keep the NOD RPZ zone up-to-date by doing incremental zone transfers, we recommend using zone transfers. Doing zone transfers ensures you always have the freshest possible NOD RPZ data.
The maintainer of rbldnsd recommends use of rsync, so we offer rbldnsd for those running rbldnsd. NOD rbldnsd files are updated once per minute.
Q. I-6) What nameserver software is compatible with NOD RPZ?
A list of nameserver software products able to ingest RPZ files can be found at the DNS RPZ website.
As of April 29th, 2016, that list consists of BIND 9, BlueCat DNS, PowerDNS, Knot and InfoBlox DNS Firewall.
Q. I-7) NOD RPZ is currently offered with coverage thresholds of 5 minutes, 10 minutes, 30 minutes, 60 minutes, 3 hours, 12 hours, and 24 hours. Which of those seven fixed time intervals should I select?
The range of coverage thresholds offered are designed to provide a range of protection options that should meet the needs of most sites, with intervals that line up with typical filter update frequencies. If you’re not sure what time coverage will give you the most appropriate coverage, we suggest you start with one hour.
Q. I-8) Am I allowed to retain NOD data for longer than the specified period, if, for example, I need an archival record of what was listed on NOD?
Yes, provided your actions are consistent with the terms and conditions of your contract with Farsight. If you have specific questions about any contemplated use, please contact your account representative.
Q. I-9) If I license NOD primarily for spam protection, can I also use that data for other purposes, too?
Yes, provided your actions are consistent with the terms and conditions of your contract with Farsight. If you have specific questions about any contemplated use, please contact your account representative.
S. II) Technical Questions
Q. II-1) I’ve already got pretty good spam protection through use of various blocklists and SpamAssassin heuristics. Why do I need NOD, too? How much will it really help?
In a nutshell, even with state-of-the-art spam filtering, some spam still ends up getting delivered to user inboxes at most sites. Using NOD can potentially drive that residual spam down still further. Farsight has discussed this in a number of Blog posts. Please see:
Q. II-2) I’m seeing some domains show up in NOD that I think we’ve already seen… what’s going on?
When this question has come up from time-to-time in the past, invariably we’ve found that the domains in question are ones that are correctly listed.
The confusion is typically associated with the fact that the domains that were noticed are ones associated with effective TLDs rather than real TLDs. Farsight use the Public Suffix List to help us recognize times some domains require special handling. For example, consider:
r3e1r1s80yak6y5ataiaw-20jlj9qygg93j295u1j6yzatql-1368890819-eus.cloudapp.net.
r221fzc0jq1g9ynhpv3d3n-0e4baztzcyr4b2rxwkaynx0dhm-604296285-wus.cloudapp.net.
r5a1y0nfiuhbph2n1dv2-2ys4n3mn5lgfn23k9mc6gdgirx-1357037440-ncus.cloudapp.net.
r321sv4ubmup44pg9rk2-0dep440wpml202g1t2jkm01h9e-1368890819-ncus.cloudapp.net.
r4a1qtzmfsuju5lpyyk4f-3cpgikz27666920fu69lp3qk0g-357175217-eus2.cloudapp.net.
All of those were newly observed domains reported on Channel 212, notwithstanding the fact that all those domain (and many more!) share the common suffix of “cloudapp.net.” So why list those domains? The answer is simple: cloudapp.net is listed as an effective TLD in Public Suffix and we’d never previously seen any of those domains previously, considering the entire string shown above.
Q. II-3) Does an observation by a Farsight sensor of any sort of DNS record type “count” for a “first observation” or do only some types of records (such as “A” records) “count?”
The first observation of any sort of DNS record “counts.”
Q. II-4) How long is the delay between the time a domain is seen by a Farsight sensor and the time that domain shows up in a customer’s feed?
Approximately 30 seconds.
Q. II-5) How often do the NOD data files get updated? How often can we sync new copies of the NOD data?
The NOD RPZ files are updated continuously with DNS NOTIFY packets being sent at least 5 seconds apart.
The NOD rbldnsd files are written every 10 seconds but we request that you limit your synchronizations to once per minute.
Q. II-6) How many Newly Observed Domains are seen per second or minute or day? How many Newly Observed Hostnames?
The observed number of Newly Observed Domains and Newly Observed Hostnames will vary from day-to-day, but as an example, here are graphs for both Newly Observed Domains (Channel 212) and Newly Observed Hostnames (Channel 213) for the week prior to April 29th, 2016. Both graphs have had internal hostname references redacted so those graphs could be included in this writeup. (And note the different Y-axis scales!).
Channel 212 (NOD) averaged 1.25 new domains per second, or about 108,000 per day.
Channel 213 (NOH) averaged 126.12 new domains per second, or about 10.9 million per day
Q. II-7) Can I retrospectively see exactly when a Newly Observed Domain was first observed, even if it has already expired from the NOD fed?
Yes, check DNSDB (another separately-licensed product) for the domain of interest. DNSDB will report the time of first observation.
S. III) Provisioning Questions
Q. III-1) Why do you need to know our IP address ranges? If we have a couple of IPv4 /16’s, can I just give you the addresses of those netblocks?
We use network access control lists (ACLs) to limit access to our servers. We need to explicitly permit authorized customers to access our boxes, and we can only do that if we know the public IPs your servers are using.
To keep the holes in our firewall to a minimum, we ask that you list only the IP addresses necessary for your servers to communicate — please do NOT just list all your address space by default (we can live with an IPv4 /24 or two, but a couple of IPv4 /16’s would just be too broad).
Q. III-2) Are any “holes” needed in my firewall for inbound traffic if I’m using NOD RPZ? If so, for what IPs and what ports?
Most NOD traffic is “pulled” by the customer, rather than “pushed” by Farsight.
However, if you’re trialing or have purchased NOD RPZ or DNSBL delivered via zone transfers, we need to be able to send IXFR notifications to your nameservers. Please permit 53/UDP and 53/TCP inbound from [need hostname here] to your nameserver.
Q. III-3) What’s “TSIG” and what role does it play when it comes to updates via IXFR?
TSIG, coauthored by Farsight’s very own Paul Vixie, is used to secure the incremental zone transfers Farsight does with your system. We support MD5 (now deprecated by the industry, but available for compatibility), SHA-256, SHA-384 or SHA-512 keys for that purpose.
Q. III-4) Can I run NOD on multiple nameservers for my users? (I’m looking for redundancy and/or load balancing)
Yes. NOD RPZ and NOD rbldnsd is contracted per covered user, not by name server. As long as you’re not protecting more users than authorized, and you respect the provisions of your contract with Farsight, you can load NOD data on multiple servers for redundancy or load balancing purposes. We suggest that for best performance you slave the zone to a master server that you control and then redistribute to your other name servers.
Q. III-5) Do I need to limit access to my nameservers to ensure that unauthorized third parties don’t get access to NOD data through my servers?
Yes, consistent with your contract and non-disclosure agreement, you must limit access to your nameservers so as to only allow access by licensed users.
Q. III-6) Can I get NOD via IPv6 transport?
Yes. Farsight offers both IPv4 and IPv6 transport for all of its network-delivered products.
S. IV) Pricing/Licensing Questions
Q. IV-1) How much does NOD cost?
To request a demonstration of NOD contact the DomainTools sales team.
