DomainTools Announces Predictive Threat Feeds - Powering Preemptive Exposure Management

Farsight TXT Record

Farsight's Network Message, Volume 4: The C Programming API

Written by:

Farsight Security

Published on:

Feb 18, 2015

On This Page

TOC Element

Abstract

This article is the fourth in a multi-part blog series intended to introduceand acquaint the user with Farsight Security’s NMSG suite. This articleintroduces the

libnmsg

C programming API.

Before reading this article, it is recommended that you read the followingFarsight Security Blog articles:

This article, geared towards intermediate-level C programmers, is by no meansan exhaustive API reference. For that, the reader is directed to theaccompanying Doxygen-based API manual.This article covers NMSG (protocol) version

2

and

nmsg

(C library)version

0.9.1

The reader is also directed to explore the examples directory in the

nmsg

repository, whereseveral other sample

libnmsg

-based programs will be found.

Hello World, NMSG-style

To demonstrate the some of

libnmsg

‘s functionality, we will examine a simpleexample of how to instantiate an

nmsg

session and plumb data through it. TheC source code for this program,

nmsgpacket

, is detailed below. All of thecode is contained in a single source file that can be compiled into a fullyfunctional program. To build

nmsgpacket

, you’ll need to link against

libnmsg

and

libpcap

. As such, you’ll want to ensure recent versions of bothlibraries are installed in a standard system location (such as

/usr/local/lib

nmsgpacket

nmsgpacket

is a simple program that reads network packets, converts them intoNMSG, and mirrors the payloads to a binary file and to stdout. It requires twocommand-line arguments: the network interface from which to capture packets anda count of the number of packets to capture and convert to NMSG payloads (eachcaptured packet will be converted into a single NMSG payload).

To wit:

$ ./nmsgpacket en0 5 [159] [2015-02-16 20:42:47.295350000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=154> [159] [2015-02-16 20:42:47.296414000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=154> [82] [2015-02-16 20:42:48.831159000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=78> [333] [2015-02-16 20:42:48.832891000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=328> [82] [2015-02-16 20:42:49.138423000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=78>

We use

nmsgtool

to verify the binary file’s payload contents are the sameas those display on the console:

$ nmsgtool -r nmsgpacket.nmsg [159] [2015-02-16 20:42:47.295350000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=154> [159] [2015-02-16 20:42:47.296414000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=154> [82] [2015-02-16 20:42:48.831159000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=78> [333] [2015-02-16 20:42:48.832891000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=328> [82] [2015-02-16 20:42:49.138423000] [1:12 base packet] [00000000] [] [] payload_type: IP payload: <BYTE ARRAY LEN=78>

nmsgpacket.c

The following is the entire 193 line C source code file with in-lineannotations explaining

libnmsg

API calls.

The full source code is available for download fromFarsight Security’s blog-code GitHub page.

Preamble

The first section contains the source code license and the standard C headerfile include progression:

/* * Copyright (c) 2015 by Farsight Security, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ #include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <errno.h> #include <unistd.h> #include <pcap.h> #include <nmsg.h> #include <nmsg/base/defs.h>

Declare NMSG objects

Outlier

nmsg

-specific variables include the following:

nmsg_io_t io

: The opaque

nmsg

io engine object. It handles multiplexing of NMSG data between inputs and outputs.

nmsg_res res

: The

nmsg

result code object. It is used by the

libnmsg

API to return meaningful result codes that can be looked up with

nmsg_res_lookup()

nmsg_input_t input

: The opaque

nmsg

input object. It is used by

nmsg

to stream payloads from different input sources.

nmsg_output_t output

: The opaque

nmsg

output object. It is used by

nmsg

to buffer and write payloads to output destinations.

nmsg_msgmod_t mod

: The opaque message module object. It holds message module specific state.

int main(int argc, char **argv) { int fd, rc, count; nmsg_io_t io; nmsg_res res; pcap_t *phandle; nmsg_pcap_t pcap; nmsg_msgmod_t mod; nmsg_input_t input; nmsg_output_t output; char errbuf[PCAP_ERRBUF_SIZE];

Verify command-line arguments

nmsgpacket

requires two command-line arguments, the network interface andpacket count. Anything more or less and it will throw an error:

if (argc != 3 || ((count = atoi(argv[2])) < 1)) { printf("%s capture packets and encode as NMSG base:packet\n", argv[0]); printf("usage: %s interface count\n", argv[0]); printf(" interface:\tlook for packets here\n"); printf(" count:\tprocess this many payloads (positive integer)\n"); return EXIT_FAILURE; }

Initialize libnmsg

Before any

libnmsg

API functions can be called, the library has to beinitialized:

/* initialize nmsg */ res = nmsg_init(); if (res != nmsg_res_success) { fprintf(stderr, "nmsg_init(): failed: %s\n", nmsg_res_lookup(res)); return EXIT_FAILURE; }

Initialize the IO engine

The program employs a key feature of

libnmsg

, the multi-threaded IO engine.It handles the multiplexing of data between inputs and outputs transparentlyfor the application programmer and makes

libnmsg

programming much easier. Itis initialized here:

/* initialize the nmsg io engine */ io = nmsg_io_init(); if (io == NULL) { fprintf(stderr, "nmsg_io_init(): failed\n"); return EXIT_FAILURE; }

Lookup message module

nmsgpacket

will be sourcing input as packets from a network interface. Assuch, it will use the NMSG

base

packet

message module. It is looked uphere:

/* get the message module for base:packet */ mod = nmsg_msgmod_lookup(NMSG_VENDOR_BASE_ID, NMSG_VENDOR_BASE_PACKET_ID); if (mod == NULL) { fprintf(stderr, "nmsg_msgmod_lookup(): unknown msgmod\n"); return EXIT_FAILURE; }

Initialize libpcap

The next block of code is straightforward and mostly boilerplate

libpcap

initialization, using the newer style interface API. Noteworthy here is thesnapshot length of

NMSG_DEFAULT_SNAPLEN

(recommended by the

libnmsg

API documentation) and the timeout of 1000 ms (according to

libpcap

APIdocumentation, not setting a timeout can result in undefined behavior.):

/* initialize a pcap handle */ phandle = pcap_create(argv[1], errbuf); if (phandle == NULL) { fprintf(stderr, "pcap_create (%s): %s\n", argv[1], errbuf); return EXIT_FAILURE; } /* set the pcap snapshot length to the nmsg recommended value */ rc = pcap_set_snaplen(phandle, NMSG_DEFAULT_SNAPLEN); if (rc != 0) { fprintf(stderr, "pcap_set_snaplen() failed: %d\n", rc); return EXIT_FAILURE; } /* set the pcap read timeout to 1000ms */ rc = pcap_set_timeout(phandle, 1000); if (rc != 0) { fprintf(stderr, "pcap_set_timeout() failed: %d\n", rc); return EXIT_FAILURE; } /* start packet capture */ rc = pcap_activate(phandle); if (rc != 0) { fprintf(stderr, "pcap_activate() failed with error code: %d\n", rc); return EXIT_FAILURE; }

NMSG pcap input

libnmsg

provides a reassembled IP datagram interface to

libpcap

. Thisinterface is quite handy for

libnmsg

programmers that write code to collectnetwork packets but don’t want to have to deal with the minutia of handling IPfragments.

nmsgpacket

uses this interface and opens a new

libnmsg

pcapinput from the previously opened

libpcap

handle. It then instantiates aninput object:

/* initialize a new nmsg pcap input */ pcap = nmsg_pcap_input_open(phandle); if (pcap == NULL) { fprintf(stderr, "nmsg_pcap_input_open() failed\n"); return EXIT_FAILURE; } /* initialize a new NMSG pcap input from a pcap descriptor */ input = nmsg_input_open_pcap(pcap, mod); if (input == NULL) { fprintf(stderr, "nmsg_input_open_pcap(): failed\n"); return EXIT_FAILURE; }

Add pcap input

The input object is then attached to the IO engine:

/* add an nmsg input object to the io engine */ res = nmsg_io_add_input(io, input, NULL); if (res != nmsg_res_success) { fprintf(stderr, "nmsg_io_add_intput() failed: %s\n", nmsg_res_lookup(res)); return EXIT_FAILURE; }

NMSG binary output

Next, the program sets up the binary output object. A file is opened and thefile descriptor is used to instantiate a binary file output object which isthen attached to the IO engine:

/* open a new file to write binary NMSGs */ fd = open("nmsgpacket.nmsg", O_WRONLY | O_CREAT | O_TRUNC, 0644); if (fd == -1) { fprintf(stderr, "open() failed: %s\n", strerror(errno)); return EXIT_FAILURE; } /* initialize a new binary file nmsg output */ output = nmsg_output_open_file(fd, NMSG_WBUFSZ_MAX); if (output == NULL) { fprintf(stderr, "nmsg_output_open_file(): failed\n"); return EXIT_FAILURE; } /* add an nmsg output object to the io engine */ res = nmsg_io_add_output(io, output, NULL); if (res != nmsg_res_success) { fprintf(stderr, "nmsg_io_add_output() failed: %s\n", nmsg_res_lookup(res)); return EXIT_FAILURE; }

NMSG presentation output

Next, the same process is used for a presentation output object (withoutopening a file —

stdout

is already open). The presentation output objectis attached to the IO engine:

/* initialize a new stdout presentation format nmsg output */ output = nmsg_output_open_pres(STDOUT_FILENO); if (output == NULL) { fprintf(stderr, "nmsg_output_open_pres(): failed\n"); return EXIT_FAILURE; } /* add an nmsg output object to the io engine */ res = nmsg_io_add_output(io, output, NULL); if (res != nmsg_res_success) { fprintf(stderr, "nmsg_io_add_output() failed: %s\n", nmsg_res_lookup(res)); return EXIT_FAILURE; }

Tune the IO engine

First, the IO engine is configured to stop after processing the user specifiednumber of payloads. Next, it is configured to mirror payloads across eachoutput. Important to note here is that mirroring imposes the overhead of aper-output copy for each input payload (striping is default mode of operation).

/* configure the io engine to close after count payloads */ nmsg_io_set_count(io, atoi(argv[2])); /* configure the io engine to mirror payloads to each output */ nmsg_io_set_output_mode(io, nmsg_io_output_mode_mirror);

Run the IO loop

Finally, the IO engine is started, which will read from the pcap input andmirror NMSG payloads to both outputs. After

count

payloads are processed,the IO engine is shutdown and resources are freed.

/* start processing nmsg inputs and outputs */ res = nmsg_io_loop(io); if (res != nmsg_res_success) { fprintf(stderr, "nmsg_io_loop() failed: %s\n", nmsg_res_lookup(res)); return EXIT_FAILURE; } /* deallocate the resources associated with the io engine */ nmsg_io_destroy(&io); return EXIT_SUCCESS; }