Farsight TXT Record

NMSG and JSON encoding

Written by:

Farsight Security

Published on:

May 6, 2015

Abstract

This article introduces

njt

(amalgamation of NMSG+JSON+tool), a newconvenience tool used for working with

base

encode

(JSON) NMSGs at thecommand line. With this tool, the user has a previously unavailablecommand-line interface to serialize arbitrary JSON as NMSG

base

encode

(JSON) protocol data units (PDUs) or de-serialize

base

encode

(JSON) NMSG PDUs to JSON.

To get the most from this article, it is recommended that you be comfortablewith the material from the following Farsight Security Blog articles:

Introduction

Farsight Security’s

nmsgtool

isthe de facto tool for sending and receiving NMSGs. To encode and decodedifferent message types, it employs a run-time loadable module based system.These modules provide an extensible framework of protobuf encoders anddecoders. While

nmsgtool

certainly has a

base

module for encoding anddecoding

base

encode

messages, it does not include support for decodingthe “last mile” of whatever data was encoded in the first place.

In fact, the

base

encode

module was provided as a stopgap for users who wishto encode new data, but don’t want to deal with the heavy lifting of writing anew protobuf module.

Until now, if you wanted to get at the data inside a

base

encode

NMSG,you had to write your own program to decode it using theC APIor thePython API. Choosing what we felt was the most popular and most widely used formatwe wrote

njt

to provide our users with a simple and singularinterface to manage encoding JSON to NMSG and decoding of NMSG data into JSON.

Use Cases

njt

was developed as a convenience for

nmsgtool

users as well asFarsight DNSDB,SIE, and SRA customers. Some of the more common use casesthat are readily available with

njt

include:

Encode a pre-existing JSON file into an on-disk compressed NMSG file for convenient storage / subsequent transmission
Decode on-disk NMSGs containing JSON-encoded data
Decode live JSON-encoded SIE Channel data
Encode DNSDB JSON-encoded query responses as NMSG and stream to a remote nmsgtool endpoint

Each will be detailed below.

JSON Record Delineation

There are several ways to separate multiple JSON objects in a stream, some ofthe more common methods are described below (Thanks to Robert Edmonds fortaking the time to bikeshedthis with me).

A top-level JSON array that contains each object.
“newline-delimited JSON” (also known as “JSON Lines”). This format uses

\n

delimiters between objects and bans the use literal

\n

characters in JSON objects. Newline-delimited JSON is a convenient and natural format for storing structured data that may be processed one record at a time. This is what the DNSDB API uses as well as certain SIE channels.
“newline delimited” JSON, with

\r\n

delimiters between objects, allowing literal

\n

characters in JSON objects. This is what the Twitter streaming API uses.
The “json-seq” MIME type, which uses ASCII “record separator”

0x1E

and line feed characters to separate JSON objects. This is standardized in RFC 7464.

Currently, the only type of JSON

njt

expects and emits is newline-delimitedJSON. Each record is expected to be terminated with a literal

\n

and no

\n

‘s can appear inside the JSON. If there is demand to process other forms ofrecord separation, Farsight Security will add support in future releases.

njt Download

njt

is an open-source Python tool available for download from Farsight Security’s GitHub repository here.

Usage

Invoked with

--help

njt

dumps the following usage message:

$ ./scripts/njt --help usage: njt [-h] (-e | -d) [-c COUNT] [-w OUT_FILE] [-p] [-z] [--setsource SETSOURCE] [-V] [-v] [--setoperator SETOPERATOR] [--setgroup SETGROUP] [in_file] Serialize JSON as base:encode(JSON) NMSG PDUs or deserialize base:encode(JSON) NMSG PDUs to JSON positional arguments: in_file input file, also accepts input from pipeline optional arguments: -h, --help show this help message and exit -e, --encode encode JSON --> NMSG -d, --decode decode NMSG --> JSON -c COUNT, --count COUNT stop after count payloads -w OUT_FILE, --out_file OUT_FILE write NMSG data to file -p, --prettyprint sort and pretty print JSON output -z, --zlibout compress NMSG output --setsource SETSOURCE set payload source -V, --verbose print debugging information -v, --version show program's version number and exit --setoperator SETOPERATOR set payload operator --setgroup SETGROUP set payload group

njt

must be invoked in one of two modes, either

-e

to encode JSON to NMSGor

-d

to decode NMSG to JSON. It accepts input using all of the standardeverything-is-a-file paradigm:

Positional file argument at command line: $ njt -e test.jsonl
Redirect input from a file: $ njt -d < test.nmsg
Output of a pipeline: $ cat test.jsonl | njt -e
Additionally, the encoder can read ASCII text at command line:$ njt -e^M {"count": 1}^M^D

Encode JSON file

The simplest and probably most common use case of

njt

is to encode a(previously created) JSON file into NMSG. Using the bundled 16 record test file

test.jsonl

, the invocation is as simply:

$ njt -e test.jsonl

This creates the following NMSG file:

$ ls -l njt.out* -rw-r--r-- 1 mschiffm mschiffm 4008 May 2 01:14 njt.out.1430529251.66.nmsg

We can verify the veracity of the NMSG file and count the number of payloads:

$ nmsgpcnt-test njt.out.1430529251.66.nmsg containers: 1 payloads: 16

In addition to accepting input from a pipeline,

njt

also supports several

nmsgtool

-derived options to modify the output like setting the source ID andoperator, choosing a file name and compressing the output:

$ cat test.jsonl | njt -e -V --setsource 0xdeadbeef --setoperator FSI -w test.nmsg wrote 168 byte payload wrote 233 byte payload wrote 196 byte payload wrote 222 byte payload wrote 164 byte payload wrote 163 byte payload wrote 223 byte payload wrote 185 byte payload wrote 214 byte payload wrote 231 byte payload wrote 222 byte payload wrote 219 byte payload wrote 220 byte payload wrote 220 byte payload wrote 219 byte payload wrote 220 byte payload Finished, wrote 3319 bytes in 16 payloads to test.nmsg

With

nmsgtool

, we can verify one of the NMSGs is what we expect:

$ nmsgtool -r test.nmsg -c 1 [173] [2015-05-05 18:57:36.232985973] [1:11 base encode] [deadbeef] [FSI] [] type: JSON payload:

Decode NMSGs

Using the NMSG file created above, we can easily decode and pretty-print asingle JSON record:

$ njt -d test.nmsg -p -c 1 { "bailiwick": "example.com.", "count": 2, "rdata": [ "10 foo.example.ru." ], "rrname": "example.com.", "rrtype": "MX", "time_first": 1372708329, "time_last": 1372708329 }

Additionally,

njt

supports pipelining directly into jq, a powerful command-line tool offering rich functionality forprocessing JSON data. Using it with

njt

, we can slice and filter output toour liking. For example, to emit just the

rrtypes

, we can issue thefollowing command:

$ njt -d test.nmsg | jq ".rrtype" "MX" "NS" "NS" "NS" "A" "A" "NS" "NS" "NS" "NS" "NS" "SOA" "SOA" "SOA" "SOA" "SOA"

Decode live SIE Channel data

Another useful option

njt

offers is the ability to decode live data fromFarsight Security’s SIE. Any JSON-encoded feed can be decoded and emitted. Forexample, if you are a base-channel package subscriber, you can decode Channel42 (anonymized IDS and firewall logs from ThreatSTOP):

$ nmsgtool -C ch42 -c 1 -w - | njt -d -p { "Alert": { "AdditionalData": [ { "content": "1", "meaning": "direction" }, { "content": "1", "meaning": "anon" }, { "content": "1", "meaning": "version" }, { "content": "apr 20 20:34:14 ...

Encode DNSDB queries and Stream to a Remote Endpoint

Busier pipelines are available to

njt

. Using Farsight Security’s Python DNSDBquery tool

dnsdb_query.py

and

nmsgtool

DNSDB API customers can package upDNSDB query responses and NMSG and stream them to a remote endpoint.

First, set up an

nmsgtool

listener on an unused port (for this simulationwe’ll use loopback but in practice any unfiltered IP address will work):

$ nmsgtool -l 127.0.0.1/9430

Next, issue a DNSDB query, encode the returned JSON as NMSG and use

nmsgtool

to write the payloads to the network:

$ dnsdb_query.py -r example.com -j | njt -e | nmsgtool -r - -s 127.0.0.1/9430

And

nmsgtool

emits:

[244] [2015-04-26 21:29:04.044215917] [1:11 base encode] [00000000] [] [] type: JSON payload: ...

As above, decoding and filtering are available. To do this, substitute theoriginal

nmsgtool

listener invocation for something like:

$ nmsgtool --unbuffered -l 127.0.0.1/9430 -w - | njt -d | jq ".rrtype"

Mike Schiffman is a Packet Esotericist for Farsight Security, Inc.