Farsight's Network Message, Volume 1: Introduction to NMSG
Abstract
This article is the first in a multi-part blog series intended to introduce andacquaint the user with Farsight Security’s NMSG suite. This article is anintroduction to NMSG and covers version
0.9.1
.
Why NMSG?
As you may already know, Farsight Security’s Security Information Exchange (SIE) transits a great deal of variably typed data. As a result, we needed specialized software that would enable us,our customers, and our partners to painlessly serialize and transmit this everchanging data. That specialized software is NMSG.
Going forward, note that NMSG (all caps) refers to the protocol while nmsg(lowercase) refers to the C implementation of the protocol.
Everything Comes From Something
Back before Farsight Security acquired security assets fromISC, SIE was just coming online and in an embryonic stage.It was at this time, that NMSG’s direct antecedent, NCAP (Network Capture) wasconceived and developed. As SIE initially only streamed Passive DNS data, NCAPwas designed to provide encapsulation services for DNS payload data.Unfortunately, NCAP was not extensible and as SIE grew in size and scope, a newplatform was required. From this, NMSG was developed.
NMSG was designed as a single platform to accommodate (then ISC’s, and now)Farsight’s requirement to consume, emit, and process real-time flows of manydisparate types of structured data. Taking into account the lessons learnedfrom NCAP, NMSG is modular in design allowing for future expansion.
Let’s get technical
NMSG is an extensible container wire and file format for storing andtransmitting blobs of data with support for dynamic message types, compression,fragmentation, sequencing, and rate limiting.
Breaking all of that down:
- Extensible: NMSG functionality can be extended via its dynamic messagemodule interface. As new data feeds are added to SIE, corresponding newmessage modules can be developed for nmsg that do not necessitate librarycompilation or API changes.
- Container-based: NMSG data is serialized inside containers that can containone payload, many payloads, or a fraction of a large payload.
- Wire format: NMSG specifies a wire format optimized for transmission usingUDP over jumbo Ethernet
- File format: NMSG also specifies an on-disk file format for storage of NMSGdata.
- Data blobs: A core tenet behind NMSG is data agnosticism. Some of the dataFarsight consumes, ships and stores isn’t best represented in its nativeformat as frames, packets, datagrams, segments, or other data primitives.To this end, NMSG was designed to be ignorant of the data it ferries. NMSGoffloads the details of encoding to external message modules and in factcan work with opaque containers.
- Dynamic message types: NMSG provides a dynamic interface to extend atrun-time the message types it understands. This keeps the library itselfrather generic and offloads the more exotic message handling to “load asneeded” external modules.
- Compression: NMSG supports in-band compression of payloads. This isimplemented in nmsg via zlib.
- Fragmentation: For payloads too large for the underlying transport, NMSG provides an automagic fragmentation service that is seamless to the useror application programmer.
- Sequencing: NMSG can optionally be configured to assign monotonicallyincreasing sequence numbers to the containers it emits. This can be used bythe receiving end to detect container loss.
- Rate-limiting: NMSG can optionally be configured to rate limit itsemission of containers so as not to overwhelm receivers on slower networks.
For the end user
NMSG is delivered to the application programmer as a C library called
libnmsg
. The library presents a rich API for the programmer to buildNMSG-capable applications and configure, tune, and/or tweak its many optionsand features.
The reference implementation of
libnmsg
is
nmsgtool
. It is a thin wrapperaround
libnmsg
that provides powerful NMSG functionality at the Unixcommand-line.
The nmsg package that contains both
libnmsg
and
nmsgtool
is freelyavailable for download:
Also available is a Python extension module,
pynmsg
, that enables NMSGdevelopment using the Python programming language:
Coming up
The next article in the NMSG series will discuss
nmsgtool
and provide someuseful recipes and examples. Future articles will examine low-level NMSGimplementation details such as header composition, data encoding and introducethe programming APIs.
Mike Schiffman is a Protocol Legerdemainist for Farsight Security, Inc.
Read the next part in this series: Farsight’s Network Message, Volume 2: Introduction to nmsgtool