Introduction
We are pleased to announce that we have added support for
JSON serialization to
libnmsg in version 0.10.0. This makes
it easier than ever for you to consume NMSG data from our Security Information Exchange (SIE).
Data can now be directly imported into your
Apache Hadoop, Splunk,
HP ArcSight, and
IBM Security QRadar
systems.
It is also easier for you to produce data for publication on SIE without the
need of integrating libnmsg into your application. You only need to emit
compatible Line Delimited JSON objects that can then be imported by nmsgtool.
The Format
There are four required fields in the JSON schema: time, vname, mname,
and message. time is an ISO 8601 timestamp in UTC with nanosecond precision and no time zone designator. vname
is a string corresponding to the vendor, mname is the msgtype name of the
msg module that should load the message object. The message object
contains presentation format fields specific to the msg module. Binary data is
encoded in base64 because JSON is a
unicode format and does not support the full range of binary characters.
Below is an example of a message from SIE channel 204: deduplicated domain
names.
{
"time": "2015-09-02 17:10:00.464105589",
"vname": "SIE",
"mname": "dnsdedupe",
"source": "a1ba02cf",
"message": {
"type": "EXPIRATION",
"count": 1,
"time_first": "2015-09-02 11:29:14",
"time_last": "2015-09-02 11:29:14",
"bailiwick": "191.54.in-addr.arpa.",
"rrname": "84.152.191.54.in-addr.arpa.",
"rrclass": "IN",
"rrtype": "PTR",
"rrttl": 300,
"rdata": ["server2.mometrix.com."]
}
}
Installation
Follow the SIE Installation
instructions. You will need to install wdns 0.8.0 or newer, nmsg 0.10.0 or
newer, and pynmsg 0.4.0 or newer.
You will also need to install a patched version of yajl until this
pull request is incorporated (there
exists a memory leak in the yajl tree parser library that occurs whenever
you try to parse invalid JSON strings that has nested objects or arrays. See
the linked pull request for the 3-line fix. This can also, of course, be
worked around by not parsing any invalid JSON strings).
Command line Usage via nmsgtool
We have added two command line options to nmsgtool: -j for input and -J
for output. Use it just as you would the -r and -w options for binary
format.
[-j file] or --readjson read json format data from file [-J file] or --writejson write json format data to file
Read a message from channel 204 and emit to stdout as JSON
$ nmsgtool -C ch204 -c 1 -J -
Read a JSON-formatted file and emit to stdout as presentation format
$ nmsgtool -j input.json
API Usage in C
You can encode and decode messages using the nmsg_message_from_json() and
nmsg_message_to_json() functions. Both functions set the pointer in their
second argument on success (return code of nmsg_res_success) to a block of
memory that the caller is responsible for deallocating after use.
nmsg_res nmsg_message_from_json(const char *json, nmsg_message_t *msg);
nmsg_res nmsg_message_to_json(nmsg_message_t msg, char **json);
You may also instantiate input and output objects that may be used via API
or by attaching to a nmsg_io object. Simply pass an open file descriptor
and then call using the normal nmsg_input and nmsg_output APIs.
nmsg_input_t nmsg_input_open_json(int fd);
nmsg_output_t nmsg_output_open_json(int fd);
msgmod API Changes
We have added two new functions to the msgmod field structure: format and
parse. You will need to implement them in your custom nmsg msgmods if you
want presentation formatting in the message object. Without them, libnmsg
will fall back on the standard formatting for the field’s data type.
typedef nmsg_res (*nmsg_msgmod_field_format_fp)(nmsg_message_t m, struct nmsg_msgmod_field *field, void *ptr, struct nmsg_strbuf *sb, const char *endline);
typedef nmsg_res (*nmsg_msgmod_field_parse_fp)(nmsg_message_t m, struct nmsg_msgmod_field *field, const char *value, void **ptr, size_t *len, const char *endline);
struct nmsg_msgmod_field { ..., nmsg_msgmod_field_format_fp format; nmsg_msgmod_field_parse_fp parse; ... }
Python
Usage of these new features within the Python API is similar to that of the
C API. You will need to use pynmsg 0.4.0 or later.
nmsg.message.from_json(str)
m.to_json()
The nmsg.message.from_json function takes a JSON object as a parameter and
return a nmsg.message object. Correspondingly, each message object has a
to_json function that returns the JSON reperesentation of that object.
You can instantiate input and output objects using the nmsg.input.open_json
and nmsg.output.open_json functions. They are called with file names,
integer file descriptors or file objects. The returned object can be used
as-is or attached to a nmsg.io object.
nmsg.input.open_json(...)
nmsg.output.open_json(...)
Conclusion
We hope that this new addition to the NMSG suite will make it easier for you to integrate our data more tightly with your back-end analysis platform of choice as well as make it easier to share your data over SIE.
If you have a sales question, please contact the Farsight Security Sales department at [email protected]. If you have a technical question, please reach out to our technical team.
Henry Stern is a Senior Distributed System Engineer for Farsight Security, Inc.
