Abstract
This article explains Farsight’s Network Message (NMSG) loss tracking feature.
To get the most from this article, it is recommended that you be comfortablewith the material from the following Farsight Security Blog articles:
- Farsight’s Network Message, Volume 1: Introduction to NMSG
- Farsight’s Network Message, Volume 2: Introduction to nmsgtool
- Farsight’s Network Message, Volume 3: Headers and Encoding
- Google Protocol Buffer Deserialization The Hard Way
Introduction
NMSG is an extensible container wire and file format for storing andtransmitting blobs of data with support for dynamic message types, compression,fragmentation, sequencing, and rate limiting. We will cover NMSG’s data sequencing feature.
NMSG’s data sequencing feature, referred to as “seqsrc” was added in version
0.8.0
to provide a mechanism for tracking container loss across UDP-socketbased NMSG sessions. This is especially useful when a user wants to trackpacket loss in high-throughput NMSG sessions.
The seqsrc feature is available in
nmsgtool
and exposed via the C API,examples of both are given below.
How it Works
Seqsrc loss tracking is performed by nmsg on the receiver side on aper-container basis (as opposed to per-payload). Each NMSG container carrieswith it a sequence number and sequence_id. The sequence number is an unsigned32 bit number that serves as — you probably guessed — the container’ssequence number. The sequence_id is an unsigned 64 bit nonce used as an opaque pseudo-random cookie to prevent inadvertent sequence space reuse. From the nmsg API documentation, thesequence_id is:
a randomized ID number identifying the sequence number space that the‘sequence’ parameter exists in. This ID number is used by NMSG consumers todisambiguate multiple disparate sequences of NMSG containers when consuming amultiplexed stream from multiple sources. This ID number should be generated bya cryptographically secure PRNG such as the one provided by nmsg_random_init()to minimize the possibility of collisions.
Indeed, during the development of seqsrc and prior to the addition of asequence_id, Farsight Security did actually seebirthday collisions due tothe very high UDP port number reuse on high bit-rate SIE channels.
To the NMSG consumer, seqsrc is implemented as a series of separate andindependent sequence number spaces. Each of these sequence number spaces isrelevant only to the NMSG connection that it references (there should be oneper
object on the sender-side), and only for the time that itexists. In a lossless system, this value should increase monotonically withoutgaps.
NMSG will partition a new seqsrc space using the IP source address,IP protocol, source port, and sequence_id to create a tuple that gets its ownsequence space. As such, a seqsrc key is defined as:
struct nmsg_seqsrc_key {
uint64_t sequence_id;
sa_family_t af;
uint16_t port;
union {
uint8_t ip4[4];
uint8_t ip6[16];
};
};
Internally, Farsight Security uses seqsrc to monitor container loss inreal-time on its Security Information Exchange (SIE).
Loss Tracking with nmsgtool
The reference implementation of nmsg,
nmsgtool
, supports seqsrc when invokedwith (at least) four levels of debug (
-dddd
).
To see seqsrc in action, we’ll use a corpus of channel 202 NMSGs containing just over 2,000,000payloads (we use the nmsgpcnt program to count containers and payloads) and send these,from one unbuffered
nmsgtool
instance, over the loopback interface, to anotherlistening
nmsgtool
instance.
The nmsg file we’ll use:
$ ls -l 202-2000000.nmsg
-rw-r--r-- 1 mschiffm staff 753800034 Mar 7 2015 202-2000000.nmsg
$ nmsgpcnt-fsi 202-2000000.nmsg
containers: 720
payloads: 2000481
First, we instantiate an nmsg listener on loopback/UDP port 8888 with theproper debug level and instruct it to write binary nmsgs to file:
$ nmsgtool -l 127.0.0.1/8888 -dddd -w foo.nmsg
...<debug messages omitted>...
Next, we fire off an
nmsgtool
instance with the
--unbuffered
option whichwill send each payload in its own container, as fast as it can (over theloopback interface on my several year old MacBook Pro this should result insome container loss):
$ nmsgtool -r 202-100000payloads.nmsg -s 127.0.0.1/8888 --unbuffered
We then quit the initial
nmsgtool
session (via ctrl-c) and inspect the debugoutput:
...
^Cnmsgtool: signalled break
nmsg_io: iothr=0x7faf8c6002c0 count_nmsg_payload_in=1989299
_input_seqsrc_destroy: input=0x7faf8c502e50 count_recv=1989299 count_drop=11182 (0.0056)
nmsg_io: io=0x7faf8c502fe0 count_nmsg_payload_out=1989299
nmsgtool
reveals that of the 2,000,481 containers sent, it received 1,989,299and 11,182 were lost.
Loss Tracking with the nmsg C API
The nmsg C library exposes a very simple API to enable, disable, and accessseqsrc data. The following code snippet is taken from the working sample nmsgprogram
:
First, we explicitly enable seqsrc verification on a previously instantiatednmsg input object (note that as of version
0.12.1
, nmsg enables it bydefault):
...
/* Instruct io engine to enable container loss tracking. */
res = nmsg_input_set_verify_seqsrc(input, true);
if (res != nmsg_res_success)
{
fprintf(stderr, "nmsg_input_set_verify_seqsrc(): %s\n",
nmsg_res_lookup(res));
return (EXIT_FAILURE);
}
printf("verify seqsrc enabled\n");
...
After running the nmsg io engine and (presumably) processing many NMSGs, wecull and emit the received and dropped container count:
...
/* Retrieve the total number of NMSG containers that have been
* received since the nmsg_input_t object was created. */
res = nmsg_input_get_count_container_received(input, &count);
if (res != nmsg_res_success)
{
fprintf(stderr,
"nmsg_input_get_count_container_received(): %s\n",
nmsg_res_lookup(res));
return (EXIT_FAILURE);
}
printf("\nrecevied %llu containers\n", count);
/* Retrieve the total number of NMSG containers that have been
* dropped since the nmsg_input_t object was created. */
res = nmsg_input_get_count_container_dropped(input, &count);
if (res != nmsg_res_success)
{
fprintf(stderr,
"nmsg_input_get_count_container_dropped(): %s\n",
nmsg_res_lookup(res));
return (EXIT_FAILURE);
}
...
Using the same NMSG file and unbuffered sending invocation as in the exampleabove, we see the following result:
$ ./nmsg-seqsrc -l 127.0.0.1/8888
nmsg JSON emitter / seqsrc example program
nmsg initialized
nmsg io engine initialized
nmsg socket input initialized
verify seqsrc enabled
socket input added to io engine
callback initialized
callback added to io engine
entering io loop, <ctrl-c> to quit...
...<newline delimited json output omitted>...
^C
received 1041381 containers
dropped 959079 containers
nmsgjsonseqsrc
reveals that of the 2,000,481 containers sent, it received1,041,381 and 959,100 were lost (we lost more here because the
nmsgjsonseqsrc
program deserializes each incoming NMSG as JSON, a more computationallyexpensive operation than simply writing binary NMSGs to a file).
Mike Schiffman is a personal packet shopper for Farsight Security, Inc.
Heading
