SIE NMSG User Guide
This document explains the NMSG protocol and provides an introduction to some example tools that can be used to acquire and process NMSG packets. Common use cases for the tools will also be reviewed.
NMSG is an adaptable container format that allows for consistent or variable message types. NMSG container data may be streamed to a file or transmitted as UDP datagrams. NMSG containers can contain multiple NMSG messages or a fragment of a message too large to fit in a single container. The data in an NMSG container can also be compressed. Additional capabilities include sequencing and rate-limiting. This includes:
- Adaptable: NMSG functionality can be modified to meet the needs of new data formats using its adaptable message module interface. As new data feeds are added to SIE, related message modules can be developed for nmsg that do not necessitate library compilation or API changes.
- Container-Based: NMSG data is serialized inside containers that can contain one payload, many payloads, or the fragment of payload too large to fit in a single container.
- Wire Format: NMSG specifies a wire-format optimized for transmitting UDP datagrams over jumbo Ethernet.
- File Format: NMSG also specifies an on-disk file-format for storage of NMSG data.
- Container Data: A core principle of NMSG is data compatibility. Some of the data Farsight consumes, transmits, and stores is inadequately represented in its native format (such as frames, packets, datagrams, segments, or other data formats). As such, NMSG was designed to be ignorant about the data it transports. NMSG offloads the details of encoding to external message modules and can also work with opaque containers.
- Dynamic Message Types: NMSG provides an adaptable message module interface that can be modified at run-time for message types it understands. This ensures the library is generic and offloads the more unique message handling to external modules that can be loaded as-needed.
- Compression: NMSG supports per-payload compression. This is implemented in nmsg using zlib.
- Fragmentation: Payloads that are too large to fit in a single container for the underlying transport, NMSG provides a fragmentation capability that is seamless to the user or application programmer.
- Sequencing: NMSG can optionally be configured to assign sequentially increasing numbers to the containers it emits. This can be used by the receiving end to detect potential container loss.
- Rate-Limiting: NMSG can optionally be configured to rate-limit its emission of containers to ensure receivers on slower networks are not overwhelmed.
NMSG is available for the application programmer as a C library called libnmsg. The library offers a complete API for the programmer to build NMSG-capable applications and configure, tune, and/or tweak its many options and features.
The reference implementation of libnmsg is nmsgtool, which is a thin wrapper
around the libnmsg C library. nmsgtool provides comprehensive NMSG
functionality at the command line interface (CLI) for Unix-like systems.
To ensure development is easy in Python and Perl, modules also exist for each programming language.
Note: Using NMSG requires that Farsight Security has provisioned at least one of the following Access Methods to acquire data from SIE:
- SIE Remote Access (SRA) (Also known as Advanced Exchange Access (AXA))
- SIE Direct Connect
- SIE Blade Server leased from Farsight
- SIE Batch
Introduction
nmsgtool is a CLI tool for the libnmsg library and is a wrapper around
libnmsg‘s input/output (I/O) engine. libnmsg controls the transmission,
storage, creation, and conversion of NMSG payloads.
nmsgtool‘s primary purpose is for prototyping and debugging access to NSMG
container format data. While it provides comprehensive NMSG functionality and
can acquire and store data from SIE, it also lacks some advanced features of
libnmsg.
NMSG Inputs and Outputs
The nmsgtool program is a single tool for acquiring a variety of different inputs, like data streams from the network, capturing data from network interfaces, reading data from files, or even standard input and making NMSG payloads available to one or more outputs. The outputs are files in binary or human-readable (ASCII presentation) form, or binary payloads to network sockets for transport. Without having to create a program for each function, nmsgtool handles all types of data processing which includes serialization, fragmentation, compression, striping or mirroring, rolling file outputs, and executing data processing programs on file outputs.
The nmsgtool program is a single tool for acquiring a variety of different inputs, like data streams from the network, capturing data from network interfaces, reading data from files, or even standard input and making NMSG payloads available to one or more outputs. The outputs are files in binary or human-readable (ASCII presentation) form, or binary payloads to network sockets for transport. Without having to create a program for each function, nmsgtool handles all types of data processing which includes serialization, fragmentation, compression, striping or mirroring, rolling file outputs, and executing data processing programs on file outputs. nmsgtool inputs can take the following forms:
- File that contains binary NMSG data, which could have been created nmsgtool
- A socket that is configured to transport binary NMSG data
- IP packets in packet capture library (libpcap) file (also known as “PCAP”)
- IP packets acquired from a network interface
- A file containing ASCII presentation data
nmsgtool outputs can take the following forms:
- Binary NMSG data stored in a file
- Binary NMSG data sent to a network socket
- ASCII presentation data in a file and including standard output (stdout)
- You can specify more than one of each
Features
nmsgtool is multithreaded
libnmsg has a feature that enables you to create user-specified Dynamic Shared Objects (DSOs) for inline filtering of network messages, which may be quicker than post filtering the data. nmsgtool can load these filtering DSOs using the -F or –filter CLI argument. Documentation about this feature is available at:
https://github.com/farsightsec/nmsg/blob/master/nmsg/io.h#L139-L206
https://github.com/farsightsec/nmsg/blob/master/fltmod/nmsg_flt_sample.c
Limitations
nmsgtool does not provide easy access to seqsrc NMSG container data.
Considerations
Performance and potential container loss may be dependent on, but is not limited to, the following. All may be potential reasons for nmsgtool to throttle and drop containers.
- Transmitting NMSG data over the Internet
- Network bandwidth
- Congestion between the transmitting instance of nmsgtool and program on the system receiving the NMSG data
- Host system or receiving system
- Disk
- Memory
- CPU
- Network interface
Examples
The following examples and use-cases demonstrate the functionality of nmsgtool.
Display Data when Physically Connected to SIE
A common nmsgtool use-case for SIE customers is to acquire real-time data from
an SIE channel and display the NMSG data on the screen.
The following invocation of nmsgtool reads one (1) NMSG payload [-c 1] from
SIE Channel 212, Newly Observed Domains (NOD), [-C ch212] and emits it to
stdout as ASCII presentation data [-o -].
$ nmsgtool -C ch212 -c 1 -o -
[72] [2015-02-03 13:35:29.678474903] [2:5 SIE newdomain] [a1ba02cf] [] []
domain: s47rbh.xyz.
time_seen: 2015-02-03 13:33:20
rrname: s47rbh.xyz.
rrclass: IN (1)
rrtype: NS (2)
rdata: ns1.51dns.com.
rdata: ns2.51dns.com.Note: If no outputs are specified, ASCII presentation to stdout [-o -] is
the default behavior of nmsgtool. In examples that follow, [-o -] will be
omitted.
nmsgtool uses the nmsgtool.chalias configuration file to determine the
proper network interface to collect data from. This configuration file contains
the SIE channel number, IP address, and UDP port mappings. When a channel
number is specified on the command line, nmsgtool looks it up in the
nmsgtool.chalias file and listens on the specified network interface.
The header from an NMSG datagram is the first line displayed for an NMSG message. Breaking this down, the header fields for SIE Channel 212 (Newly Observed Domains) follow:
- [72]: Message size in bytes
- [2015-02-03 13:35:29.678474903]: UTC timestamp with nanosecond resolution
- [2:5 SIE newdomain]: Vendor and message ID, vendor and message type
- [a1ba02cf]: Optional source identifier
- []: Optional operator code
- []: Optional group code
The message payload contains key-value pairs that conform to a schema specified
by the vendor for the message type. In the preceding example, the vendor is
SIE and the message type is newdomain. nmsgtool includes dynamically
loadable modules that enable it to display the data as you see in the preceding
example and also enable NMSG-based programs or scripts to load the key-value
pairs into structures. These concepts will be explained in more detail in
future NMSG articles.
Acquire Data from SIE and Write to Binary NMSG Files
Another use-case for SIE customers is to acquire real-time data from an SIE channel and save it to the local filesystem in a binary NMSG file for analysis at a later time/date.
The following invocation of nmsgtool acquires 100,000 NMSG payloads [-c 1000000] from SIE Channel 208, DNSDB Verified Data (deduplicated, verified,
and prior to filtering), [-C ch208] and writes them to a binary NMSG file [-w ch208.nmsg]. The result is a file approximately 15.25 megabytes in size.
$ nmsgtool -C ch208 -c 100000 -w ch208.nmsg
$ stat -c "%n %s" ch208.nmsg ch208.nmsg 15246581The binary file can then be read by nmsgtool using [-r ch208.nmsg] and it
will display one (1) NMSG payload [-c 1] containing a dnsdedupe message.
The message is emitted to stdout in ASCII presentation data.
$ nmsgtool -r ch208.nmsg -c 1
[72] [2015-02-01 00:07:53.596907788] [2:1 SIE dnsdedupe] [a1ba02cf] [] []
type: EXPIRATION
count: 2
time_first: 2015-01-31 07:29:37
time_last: 2015-01-31 07:29:37
bailiwick:
rrname:
rrclass: IN (1)
rrtype: A (1)
rrttl: 43200
rdata: <redacted>NMSG Payload Compression
nmsgtool can optionally compress the payload of each NMSG container using zlib compression (the same algorithm used by gzip), either to a file or for NMSG data transmitted across the network.
Note: Compression is performed on each payload in an NMSG container, not across the entire file.
To demonstrate the on-disk storage benefit that compression provides, we can compress the data acquired in the previous example.
The following invocation reads the binary file [-r ch208.nmsg] from the previous
example, compresses each payload using [-z], and then writes the output to a new file
[-w ch208z.nmsg]. The result is a file approximately 6.43 megabytes in size, which is a
58% decrease in file size.
$ stat -c "%n %s" ch208.nmsg
ch208.nmsg 15246581
$ nmsgtool -r ch208.nmsg -w ch208z.nmsg -z
$ stat -c "%n %s" ch208z.nmsg
ch208z.nmsg 6428829Kicker Scripts and Rotating Output Files
Another useful capability that nmsgtool provides is the ability to perform
automatic file rotation (rolling) based on a duration of time or payload count.
Additionally, the user can specify a kicker script or command to run on output
files.
An example shell script follows. The script, count.sh, counts the number of
dnsdedupe payloads from a binary NMSG file using grep -c.
#!/bin/sh
echo "$1: " `nmsgtool -r "$1" | grep -c "\[2:1 SIE dnsdedupe\]"`The following invocation of nmsgtool acquires data from SIE Channel 208,
DNSDB Verified Data (deduplicated, verified, and prior to filtering), [-C ch208] and writes compressed payloads [-z] to a binary NMSG file [-w ch208]
that is prefixed with ch208. Every two (2) seconds [-t 2] the file is
closed, rotated, and the kicker script [-k count.sh] is run on the output
file. The output from each count.sh invocation is the filename followed by
the number of NMSG payloads that each file contains. The count.sh script is
invoked in the following example:
$ nmsgtool -C ch208 -w ch208 -t 2 -z -k count.sh
./ch208.20150202.0110.1422839406.364843292.nmsg: 49404
./ch208.20150202.0110.1422839408.013136741.nmsg: 80446
./ch208.20150202.0110.1422839410.024261700.nmsg: 91067
./ch208.20150202.0110.1422839412.024284315.nmsg: 86070
./ch208.20150202.0110.1422839414.033887391.nmsg: 85490
./ch208.20150202.0110.1422839416.014162500.nmsg: 90793Note: The -k cmd or --kicker cmd arguments make [-t] and [-c]
continuous. In this mode, output file names are suffixed with a timestamp and
nmsgtool runs continuously, rotating output files as payload counts are
reached or a specified duration of time expires.
Transport NMSG Data Across the Network
nmsgtool can transport NMSG payloads across an IP network to either a unicast
or broadcast address IPv4 address or a unicast IPv6 address.
For this example, two (2) nmsgtool sessions on run on two (2) separate
systems. On the receiving system, we run nmsgtool as follows:
System Receiving NMSG Payloads
The following invocation of nmsgtool listens on a network socket for NMSG
payloads sent to IPv4 address 10.0.1.52 on UDP port 9430 [-l 10.0.1.52/9430].
When NMSG payloads are observed, they will be displayed as ASCII presentation
data to stdout.
$ nmsgtool -l 10.0.1.52/9430System Sending NMSG Payloads
The following invocation of nmsgtool reads two (2) NMSG payloads [-c 2] from
the binary NMSG file created in the previous example [-r ch208...]. The NMSG
payloads are then sent to IPv4 unicast address 10.0.1.52 on UDP port 9430 [-s 10.0.1.52/9430]. On the sending system, we run nmsgtool as follows:
$ nmsgtool -r ch208.20150202.0110.1422839406.364843292.nmsg -c 2 -s 10.0.1.52/9430System Receiving and Displaying NMSG Payloads
On the receiving system, the output from one (1) of the two (2) dnsqr NMSG
payloads is displayed to stdout in ASCII presentation format. The other is
redacted and cropped for publication.
[293] [2015-02-02 01:08:21.902736000] [1:9 base dnsqr] [e9b019b8] [] []
type: UDP_QUERY_RESPONSE
query_ip:
response_ip:
proto: UDP (17)
query_port: 31211
response_port: 53
id: 7644
qname:
qclass: IN (1)
qtype: AAAA (28)
rcode: NOERROR (0)
delay: 0.182413
udp_checksum: ABSENT
[...]
[352] [2015-02-02 01:08:22.095911000] [1:9 base dnsqr] [e9b019b8] [] []
[...]Note: The system sending NMSG payloads has the option to tune network performance, including setting the NMSG container maximum transmission unit size (note this is distinct from IP MTU buffering, and rate limiting).
For more information:
Payload Striping vs Mirroring
When multiple outputs are specified, nmsgtool defaults to striping https://en.wikipedia.org/wiki/Data_striping payloads across each output. However, nmsgtool can also be configured to mirror https://en.wikipedia.org/wiki/Disk_mirroring payloads to each output.
The following invocation acquires 100 NMSG payloads [-c 100] from SIE Channel 211, Newly Active Domains, and mirrors [--mirror] the data to two (2) outputs:
- [-o -]: ASCII presentation format is sent to stdout and displayed on the screen
- [-s 10.0.1.52/9430]: Binary NMSG payloads sent to a network socket, IPv4 destination address 10.0.1.52 on UDP port 9430
$ nmsgtool -C ch211 -c 100 -o - -s 10.0.1.52/9430 --mirror
[94] [2015-02-03 08:50:19.277158975] [2:5 SIE newdomain] [a1ba02cf] [] []
[...]Acquire Data from a Network Interface or Read a PCAP File with BPF filtering
Perhaps you would like to acquire and create your own stream of NMSG payloads
sourced from live network traffic. But more specifically, you only want to
observe DNS traffic. When running nmsgtool, you can tell it to acquire IP
datagrams directly from a network interface or it can read a PCAP file.
Additionally, an optional user-defined Berkeley_Packet_Filter (BPF) can be specified to filter packets. When acquiring data from a network interface or reading a PCAP file, nmsgtool requires the user to specify the vendor and message type so it knows how to properly encode each NMSG payload.
The following invocation of nmsgtool acquires data from a network interface
[-i eth1] and filters packets for UDP port 53 [-b "udp 53"]. The NMSG
payloads are encoded as base/dnsqr [-V base] and [-T dnsqr] and emitted
in ASCII presentation format to stdout.
$ nmsgtool -i eth1 -V base -T dnsqr -b "udp 53"
[220] [2010-05-09 05:08:54.951124000] [1:9 base dnsqr] [00000000] [] []
[...]Reading from a PCAP file is syntactically similar, just substitute [-i eth1]
with [-p example.pcap].
Python module: pynmsg
Farsight is the maintainer of a Python module named pynmsg, a Python 2.7 extension module implemented in Cython for the nmsg C library. See Farsight’s Network Message, Volume 5: The Python Programming API for more information.
Introduction
Installation
- To build from source, install Python 2.7 and required dependencies.
$ sudo apt-get install python2.7 python-pip cython Install pynmsg from source code.
$ wget "https://github.com/farsightsec/pynmsg/archive/tags/v0.4.0.tar.gz" -O "pynmsg-tags-v0.4.0.tar.gz"
$ tar xzvf "pynmsg-tags-v0.4.0.tar.gz"
$ cd pynmsg-tags-v0.4.0/
$ python setup.py build
$ sudo python setup.py install
$ cd ..Install pywdns from source code.
$ wget "https://github.com/farsightsec/pywdns/archive/tags/v0.10.0.tar.gz" -O "pywdns-tags-v0.10.0.tar.gz"
$ tar xzvf "pywdns-tags-v0.10.0.tar.gz"
$ cd pywdns-tags-v0.10.0/
$ python setup.py build
$ sudo python setup.py install
$ cd ..Perl module: Net::Nmsg
Perl
Introduction
Net::Nmsg is a perl binding to libnmsg, the reference implementation of the NMSG binary structured message interchange format. See https://metacpan.org/release/Net-Nmsg for additional information.
Installation
For Debian/Ubuntu, Farsight maintains a package called libnet-nmsg-perl.
$ apt-get install libnet-nmsg-perlFor FreeBSD, Net::Nmsg is available as an official binary package.
$ pkg install p5-Net-NmsgFor other operating systems it is possible to install Net:Nmsg using CPAN.
Prior to installing, it requires libpcap development header files or libpcap
to be installed from source.
$ perl -MCPAN -e shell
cpan> install Bundle::CPAN
cpan> install Net::NmsgSome software package dependencies may ask installation questions, like hitting
[enter] when asked for a mathematic expression, or entering some minimal
system information when configuring IO::Socket.
Farsight Blog
Farsight blog articles related to SIE and NMSG.
- Farsight’s Network Message, Volume 1: Introduction to NMSG
- Farsight’s Network Message, Volume 2: Introduction to nmsgtool
- Farsight’s Network Message, Volume 3: Headers and Encoding
- Farsight’s Network Message, Volume 4: The C Programming API
- Farsight’s Network Message, Volume 5: The Python Programming API
- Processing Passive DNS With the nmsg C API
- Google Protocol Buffer Deserialization The Hard Way
- NMSG and JSON encoding
- JSON Serialization Support in nmsg
- Farsight’s Network Message: Loss Tracking Explained
Farsight blog articles related to processing SIE channels
- Finding Web Proxy Auto Discovery Protocol (WAD) related Security Exposures Using Farsight Security’s NXDOMAINs Channel
- Conficker and a Brief Tutorial on Working With SIE Channel 80 Sinkhole Data
- What (Besides NXDOMAINs) Do We See on Farsight Security’s DNS Errors Channel?
- Spotting A DNS Denial of Service Reflection Attack in SIE Darkspace Telescope Data
- Accessing Darknet Telescope Data via SIE Remote Access (SRA)
- Visualizing SIE Channel 204 Rdata
- Geolocating & Mapping IP Address Data From SIE
- Creating a Heatmap From SIE Channel 204 Data
Contacting Support
To request a demonstration of DNSDB or to inquire about a trial API key please contact the DomainTools sales team.
Appendix A – Installing NMSG Software
Compile and install from source code
Source code tarballs for the software packages below are also available from .
Example installation instructions from source on Ubuntu 16.04
Install wdns, nmsg, and sie-nmsg
- Install dependencies from Ubuntu repositories
$ sudo apt-get install build-essential pkg-config libpcap0.8-dev libprotobuf-c-dev protobuf-c-compiler libxs-dev libyajl-dev zlib1g-dev- Install
wdnsfrom source code.
$ wget "https://github.com/farsightsec/wdns/archive/tags/v0.10.0.tar.gz" -O "wdns-tags-v0.10.0.tar.gz"
$ tar xzvf "wdns-tags-v0.10.0.tar.gz"
$ cd wdns-tags-v0.10.0/
$ ./configure
$ make
$ sudo make install
$ cd ..- Install
nmsgfrom source code.
$ wget "https://github.com/farsightsec/nmsg/archive/tags/v0.15.1.tar.gz" -O "nmsg-tags-v0.15.1.tar.gz"
$ tar xzvf "nmsg-tags-v0.15.1.tar.gz"
$ cd nmsg-tags-v0.15.1/
$ ./configure
$ make
$ sudo make install
$ cd ..- Install
sie-nmsgfrom source code.
$ wget "https://github.com/farsightsec/sie-nmsg/archive/tags/v1.2.1.tar.gz" -O "sie-nmsg-tags-v1.2.1.tar.gz"
$ tar xzvf "sie-nmsg-tags-v1.2.1.tar.gz"
$ cd sie-nmsg-tags-v1.2.1/
$ ./configure
$ make
$ sudo make install
$ cd ..- Run
ldconfigto update the shared library cache.
$ sudo ldconfigAdditional information
- Farsight’s Advanced Exchange Access, part 1 of 3
- Farsight’s Advanced Exchange Access, part 2 of 3
- Farsight’s Advanced Exchange Access, part 3 of 3
- Farsight’s Advanced Exchange Access: The C Programming API, Part One
- Farsight’s Advanced Exchange Access: The C Programming API, Part Two
- Farsight’s Advanced Exchange Access: The C Programming API, Part Three
- Farsight’s Advanced Exchange Access: 1.2.0 Release Notes
- Farsight’s Advanced Exchange Access Internals: Understanding Accounting
- Farsight’s Advanced Exchange Access: The Missing Manual
