Abstract
This article is the first in a multi-part blog series intended to introduce and
acquaint the user with Farsight Security’s NMSG suite. This article is an
introduction to NMSG and covers version 0.9.1.
Why NMSG?
As you may already know, Farsight Security’s Security Information Exchange (SIE) transits a great deal of variably typed data. As a result, we needed specialized software that would enable us, our customers, and our partners to painlessly serialize and transmit this ever changing data. That specialized software is NMSG.
Going forward, note that NMSG (all caps) refers to the protocol while nmsg (lowercase) refers to the C implementation of the protocol.
Everything Comes From Something
Back before Farsight Security acquired security assets from ISC, SIE was just coming online and in an embryonic stage. It was at this time, that NMSG’s direct antecedent, NCAP (Network Capture) was conceived and developed. As SIE initially only streamed Passive DNS data, NCAP was designed to provide encapsulation services for DNS payload data. Unfortunately, NCAP was not extensible and as SIE grew in size and scope, a new platform was required. From this, NMSG was developed.
NMSG was designed as a single platform to accommodate (then ISC’s, and now) Farsight’s requirement to consume, emit, and process real-time flows of many disparate types of structured data. Taking into account the lessons learned from NCAP, NMSG is modular in design allowing for future expansion.
Let’s get technical
NMSG is an extensible container wire and file format for storing and transmitting blobs of data with support for dynamic message types, compression, fragmentation, sequencing, and rate limiting.
Breaking all of that down:
- Extensible: NMSG functionality can be extended via its dynamic message module interface. As new data feeds are added to SIE, corresponding new message modules can be developed for nmsg that do not necessitate library compilation or API changes.
- Container-based: NMSG data is serialized inside containers that can contain one payload, many payloads, or a fraction of a large payload.
- Wire format: NMSG specifies a wire format optimized for transmission using UDP over jumbo Ethernet
- File format: NMSG also specifies an on-disk file format for storage of NMSG data.
- Data blobs: A core tenet behind NMSG is data agnosticism. Some of the data Farsight consumes, ships and stores isn’t best represented in its native format as frames, packets, datagrams, segments, or other data primitives. To this end, NMSG was designed to be ignorant of the data it ferries. NMSG offloads the details of encoding to external message modules and in fact can work with opaque containers.
- Dynamic message types: NMSG provides a dynamic interface to extend at run-time the message types it understands. This keeps the library itself rather generic and offloads the more exotic message handling to “load as needed” external modules.
- Compression: NMSG supports in-band compression of payloads. This is implemented in nmsg via zlib.
- Fragmentation: For payloads too large for the underlying transport, NMSG provides an automagic fragmentation service that is seamless to the user or application programmer.
- Sequencing: NMSG can optionally be configured to assign monotonically increasing sequence numbers to the containers it emits. This can be used by the receiving end to detect container loss.
- Rate-limiting: NMSG can optionally be configured to rate limit its emission of containers so as not to overwhelm receivers on slower networks.
For the end user
NMSG is delivered to the application programmer as a C library called
libnmsg. The library presents a rich API for the programmer to build
NMSG-capable applications and configure, tune, and/or tweak its many options
and features.
The reference implementation of libnmsg is nmsgtool. It is a thin wrapper
around libnmsg that provides powerful NMSG functionality at the Unix
command-line.
The nmsg package that contains both libnmsg and nmsgtool is freely
available for download:
Also available is a Python extension module, pynmsg, that enables NMSG
development using the Python programming language:
Coming up
The next article in the NMSG series will discuss nmsgtool and provide some
useful recipes and examples. Future articles will examine low-level NMSG
implementation details such as header composition, data encoding and introduce
the programming APIs.
Mike Schiffman is a Protocol Legerdemainist for Farsight Security, Inc.
Read the next part in this series: Farsight’s Network Message, Volume 2: Introduction to nmsgtool
