JSON Serialization Support in nmsg

Introduction

We are pleased to announce that we have added support forJSON serialization tolibnmsg in version

0.10.0

. This makesit easier than ever for you to consume NMSG data from our Security Information Exchange (SIE).Data can now be directly imported into yourApache Hadoop, Splunk,HP ArcSight, and IBM Security QRadarsystems.

It is also easier for you to produce data for publication on SIE without theneed of integrating

libnmsg

into your application. You only need to emitcompatible Line Delimited JSON objects that can then be imported by

nmsgtool

.

The Format

There are four required fields in the JSON schema:

time

,

vname

,

mname

,and

message

.

time

is an ISO 8601 timestamp in UTC with nanosecond precision and no time zone designator.

vname

is a string corresponding to the vendor,

mname

is the msgtype name of themsg module that should load the

message

object. The

message

objectcontains presentation format fields specific to the msg module. Binary data isencoded in base64 because JSON is aunicode format and does not support the full range of binary characters.Below is an example of a message from SIE channel 204: deduplicated domainnames.

{
"time": "2015-09-02 17:10:00.464105589",
"vname": "SIE",
"mname": "dnsdedupe",
"source": "a1ba02cf",
"message": {
"type": "EXPIRATION",
"count": 1,
"time_first": "2015-09-02 11:29:14",
"time_last": "2015-09-02 11:29:14",
"bailiwick": "191.54.in-addr.arpa.",
"rrname": "84.152.191.54.in-addr.arpa.",
"rrclass": "IN",
"rrtype": "PTR",
"rrttl": 300,
"rdata": ["server2.mometrix.com."]
}
}


Installation

Follow the SIE Installationinstructions. You will need to install

wdns 0.8.0

or newer,

nmsg 0.10.0

ornewer, and

pynmsg 0.4.0

or newer.

You will also need to install a patched version of

yajl

until thispull request is incorporated (thereexists a memory leak in the

yajl

tree parser library that occurs wheneveryou try to parse invalid JSON strings that has nested objects or arrays. Seethe linked pull request for the 3-line fix. This can also, of course, beworked around by not parsing any invalid JSON strings).

Command line Usage via nmsgtool

We have added two command line options to nmsgtool:

-j

for input and

-J

for output. Use it just as you would the

-r

and

-w

options for binaryformat.

[-j file] or --readjson read json format data from file
[-J file] or --writejson write json format data to file


Read a message from channel 204 and emit to stdout as JSON

$ nmsgtool -C ch204 -c 1 -J -


Read a JSON-formatted file and emit to stdout as presentation format

$ nmsgtool -j input.json


API Usage in C

You can encode and decode messages using the

nmsg_message_from_json()

and

nmsg_message_to_json()

functions. Both functions set the pointer in theirsecond argument on success (return code of

nmsg_res_success

) to a block ofmemory that the caller is responsible for deallocating after use.

nmsg_res nmsg_message_from_json(const char *json, nmsg_message_t *msg);
nmsg_res nmsg_message_to_json(nmsg_message_t msg, char **json);


You may also instantiate input and output objects that may be used via APIor by attaching to a

nmsg_io

object. Simply pass an open file descriptorand then call using the normal

nmsg_input

and

nmsg_output

APIs.

nmsg_input_t nmsg_input_open_json(int fd);
nmsg_output_t nmsg_output_open_json(int fd);


msgmod API Changes

We have added two new functions to the msgmod field structure: format andparse. You will need to implement them in your custom

nmsg

msgmods if youwant presentation formatting in the

message

object. Without them,

libnmsg

will fall back on the standard formatting for the field’s data type.

typedef nmsg_res (*nmsg_msgmod_field_format_fp)(nmsg_message_t m, struct nmsg_msgmod_field *field, void *ptr, struct nmsg_strbuf *sb, const char *endline);

typedef nmsg_res (*nmsg_msgmod_field_parse_fp)(nmsg_message_t m, struct nmsg_msgmod_field *field, const char *value, void **ptr, size_t *len, const char *endline);

struct nmsg_msgmod_field { ..., nmsg_msgmod_field_format_fp format; nmsg_msgmod_field_parse_fp parse; ... }


Python

Usage of these new features within the Python API is similar to that of theC API. You will need to use

pynmsg 0.4.0

or later.

nmsg.message.from_json(str)
m.to_json()


The

nmsg.message.from_json

function takes a JSON object as a parameter andreturn a nmsg.message object. Correspondingly, each message object has a

to_json

function that returns the JSON reperesentation of that object.

You can instantiate input and output objects using the

nmsg.input.open_json

and

nmsg.output.open_json

functions. They are called with file names,integer file descriptors or file objects. The returned object can be usedas-is or attached to a nmsg.io object.

nmsg.input.open_json(...)
nmsg.output.open_json(...)


Conclusion

We hope that this new addition to the NMSG suite will make it easier for youto integrate our data more tightly with your back-end analysis platform ofchoice as well as make it easier to share your data over SIE.

If you have a sales question, please contact the Farsight SecuritySales department at [email protected]. If you have a technical question, please reachout to our technical team.

Henry Stern is a Senior Distributed System Engineer for Farsight Security, Inc.