DNSDB Export (DNSDB API On Premises) Overview
Farsight Security® Inc.’s (now a part of DomainTools) DNSDB Export (DNSDB API On Premises) is a subscription service that allows a customer to run an on-premises instance of the DNSDB API server instead of accessing the data on Farsight’s servers across the Internet.
Farsight’s passive DNS database, DNSDB, is created using passive DNS data contributed by sensor operators all around the world. Sensor data is first processed the Security Information Exchange (SIE), a cybersecurity data-sharing infrastructure owned and operated by Farsight Security, Inc prior to being inserted into DNSDB. The DNSDB database has over 100 billion unique DNS observations going back to June, 2010, and is sourced from 200,000 observations a second from Farsight’s sensor network.
About DNSDB Export
While most customers will access Farsight’s Passive DNS database by querying Farsight’s APIs over the Internet, some customers have special needs that require a local copy of the DNSDB System. DNSDB Export allows a customer to bring a copy of the data into their installation and run an instance of the DNSDB API locally.
For instance, an on-premises instance of DNSDB Export allows a customer to both access the data via the API and by direct search of the database files. In high security situations, it allows access to the data without queries crossing the Internet.
The local instance of DNSDB supports authentication through the ability to create locally issued API keys.
Because of the large size of the initial data set, for some customers, Farsight is able to ship an initial copy of the DNSDB database files using an encrypted drive to bootstrap a new installation of DNSDB Export.
Features and Benefits
DNSDB Export can be the solution for customers when:
| DNSDB Export Features | |
|---|---|
| Partitioned Network Environments | Some customers require work to be done in high security environments, including “air-gapped” systems or “partitioned” networks with no access to external networks or sites. For these customers, DNSDB Export allows all DNS analysis to be conducted without any public connectivity. |
| Privacy of Queries | For customers with a need for high security, DNSDB Export gives them an option to guarantee the security of their queries and research. Since the packets never leave the local environment, they can’t be sniffed by outside actors, guaranteeing complete privacy of their uses of the DNSDB data. |
| Direct database Access | Some customers need the ability to search the database MTBL files directly, and that requires an on-premises server. |
| Performance | For DNSDB users needing to perform tens of thousands of queries per hour, DNSDB Export allows access to the data without any network delays. |
| High Availability | DNSDB Export allows users to build fail-safe infrastructures, guaranteeing high availability of the data and avoiding outside impacts like network failures or even possible DDOS attacks. |
DNSDB Export Subscription Parameters
DNSDB data is distributed as MTBL (Immutable Sorted String Tables) files, which are read-only databases of key/value pairs organized for fast access. Files are created with different levels of granularity:
| Subscription | Available Files |
|---|---|
| DNSDB Export Minute | Yearly, Monthly, Daily, Hourly, Decaminutely, Minutely |
| DNSDB Export Hour | Yearly, Monthly, Daily, Hourly |
| DNSDB Export Day | Yearly, Monthly, Daily |
| DNSDB Export Month | Yearly, Monthly |
All DNS data appears in all of the files, as the higher granularity file data is merged into the larger data sets as appropriate. When setting up the subscription, a customer can purchase access to the appropriate level of granularity for their feed. Farsight’s support team can help define the best levels of granularity for a specific set of requirements.
DNSDB supports both DNS and DNSSEC data as separate data sets, so a customer can choose to include or exclude DNSSEC data based on their needs. DNS data is available going back to June, 2010, but a customer can choose to exclude older data from their feeds if it isn’t needed for their analysis.
DNSDB is delivered via a secure, encrypted HTTPS connection.
Hardware and Network Requirements
DNSDB Export is intended to be installed on real or virtual servers owned or controlled by the customer within a secure hosting facility owned and controlled by the customer or their service provider. These servers must be managed securely to guarantee that DNSDB data is only disclosed to authorized users.
The DNSDB data must be segregated from other data sets and cannot be merged into other databases or mingled with non-Farsight data. If the Farsight subscription is terminated then all DNSDB data must be deleted and cannot be retained.
The recommended OS for the servers is Debian 9, but these operating systems are currently supported:
- Debian 9 (Stretch) and 10 (Buster)
- Ubuntu 20 LTS (Focal Fossa)
- CentOS 7 and Red Hat 7
For complete hardware and network requirements to run a server that supports DNSDB Export, please see the document DNSDB Export Requirements.
Additional Information
- DNSDB API Documentation
- htps://github.com/dnsdb , Source repositories for DNSDB API query tools
- https://github.com/farsightsec, Source repositories for DNSDB libraries
- Farsight’s Real-time DNSDB, Part One
- Farsight’s Real-time DNSDB, Part Two
- Performing Regex Searches Against DNSDB MTBL Files
