Abstract background of spheres and lines

Farsight's Network Message, Volume 5: The Python Programming API

Abstract

This article is the fifth and final in a multi-part blog series intended to introduce and acquaint the user with Farsight Security’s NMSG suite. This article introduces the pynmsg Python programming API.

Before reading this article, it is recommended that you read the following Farsight Security Blog articles:

This article, geared towards intermediate-level Python programmers, is by no means an exhaustive API reference. This article covers NMSG (protocol) version 2 and pynmsg version 0.2.

The reader is also directed to explore the examples directory in the pynmsg repository, where several other sample pynmsg-based programs will be found.

Python and Cython

pynmsg is a Python extension module implemented in Cython for the nmsg C library. This article shows the reader how to instantiate an nmsg session using Python and then read and write NMSGs. It assumes you already have nmsg and pynmsg installed.

pynmsgpacket

pynmsgpacket is a command-line tool that processes base:packet encoded NMSGs. It winnows payloads down to those containing only TCP and UDP packets and then extracts a 5-tuple (source IP, source port, destination IP, destination address, protocol) and prints it to stdout. If so configured, the program will then use this 5-tuple to construct a new base:ipconn NMSG message and forward to a network listener.

Loyal readers will recognize pynmsgpacket builds on lessons learned from the previous NMSG blog article and sources input from files created by nmsgpacket.

First, let’s demonstrate the finished product:

    $ ./pynmsgpacket 
    usage: pynmsgpacket [-h] [-c COUNT] [-s SOCKET] [input]

    Print IPv4 TCP or UDP packet data from base:packet encoded NMSGs and
    optionally write base:ipconn NMSGs to a remote listener

    positional arguments:
      input                 input file, also accepts input from pipeline

    optional arguments:
      -h, --help            show this help message and exit
      -c COUNT, --count COUNT
                            stop after count payloads
      -s SOCKET, --socket SOCKET
                            write binary NMSGs to socket (i.e., 127.0.0.1/9430)

We can process a few payloads from a previously created nmsgpacket.nmsg file:

    $ pynmsgpacket -c 3 nmsgpacket.nmsg
    [2015-02-22 06:58:27.573210000] [1:12 base packet] [] [] [] 
    10.0.1.52.17500 --> 255.255.255.255.17500 (17)
    [2015-02-22 06:58:27.573489000] [1:12 base packet] [] [] [] 
    10.0.1.52.17500 --> 10.0.1.255.17500 (17)
    [2015-02-22 06:58:30.32007000] [1:12 base packet] [] [] [] 
    172.16.82.195.80 --> 10.0.1.51.59451 (6)

More complex command-line invocations are available as well. First, instantiate an nmsgtool listener:

    $ nmsgtool -l 10.0.1.52/9430

Then invoke nmsgtool to read from the network, encode as base:packet and pipe the NMSGs to pynmsgpacket which will dump the 5-tuple to stdout and then emit base:ipconn NMSGs to the remote listener:

    $ nmsgtool -i en0 -V base -T packet -w - --unbuffered | pynmsgacket -s 10.0.1.52/9430
    [2015-02-22 20:54:06.253410000] [1:12 base packet] [00000000] [] [] 
    10.0.1.51.61929 --> 10.0.1.52.22 (6)
    [2015-02-22 20:54:06.253410000] [1:12 base packet] [00000000] [] [] 
    10.0.1.51.61929 --> 10.0.1.52.22 (6)
    [2015-02-22 20:54:06.253410000] [1:12 base packet] [00000000] [] [] 
    10.0.1.51.61929 --> 10.0.1.52.22 (6)
    [2015-02-22 20:54:06.253418000] [1:12 base packet] [00000000] [] [] 
    10.0.1.51.61929 --> 10.0.1.52.22 (6)
    ...

At the console running the first instantiation of nmsgtool, the following is emitted:

    [20] [2015-02-22 20:54:07.143420934] [1:5 base ipconn] [00000000] [] [] 
    proto: 6
    srcip: 10.0.1.51
    srcport: 61929
    dstip: 10.0.1.52
    dstport: 22

    [20] [2015-02-22 20:54:07.143464088] [1:5 base ipconn] [00000000] [] [] 
    proto: 6
    srcip: 10.0.1.51
    srcport: 61929
    dstip: 10.0.1.52
    dstport: 22

    [20] [2015-02-22 20:54:07.143507957] [1:5 base ipconn] [00000000] [] [] 
    proto: 6
    srcip: 10.0.1.51
    srcport: 61929
    dstip: 10.0.1.52
    dstport: 22

    [20] [2015-02-22 20:54:07.143552064] [1:5 base ipconn] [00000000] [] [] 
    proto: 6
    srcip: 10.0.1.51
    srcport: 61929
    dstip: 10.0.1.52
    dstport: 22

pynmsgpacket source

The following is the entire 129 line Python source code file with in-line annotations explaining pynmsg API calls. As a matter of note, pynmsgpacket is written as a tutorial and as such doesn’t have a lot of things proper Python code should, like docstrings and try/except clauses wrapping function calls that can raise exceptions.

The full source code is available for download from Farsight Security’s blog-code GitHub page.

Preamble

The first section contains the source code license and the standard Python imports:

#!/usr/bin/env python

# Copyright (c) 2015 by Farsight Security, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# pynmsgpacket

import sys
import time
import struct
import socket
import argparse

import nmsg

Handle command-line arguments

Upon entering the main() function body, pynmsgpacket parses command-line arguments. The main thing it figures out here is whether or not to look for input NMSGs from a file or from a pipe. It also sets the count:

def main():
    parser = argparse.ArgumentParser(
            description="Print IPv4 TCP or UDP packet data from base:packet"
            " encoded NMSGs and optionally write base:ipconn NMSGs to a"
            " remote listener")
    parser.add_argument('input', nargs='?', type=argparse.FileType('r'),
                        help="input file, also accepts input from pipeline")
    parser.add_argument("-c", "--count", type=int,
                        help="stop after count payloads")
    parser.add_argument("-s", "--socket",
                        help="write binary NMSGs to socket"
                        " (i.e., 127.0.0.1/9430)")
    args = parser.parse_args()

    if not sys.stdin.isatty():
        args.input = sys.stdin

    if not args.input or args.count == 0:
        parser.print_help()
        exit(1)

    count = args.count or 0

Instantiate nmsg output object

If the user specifies -s at the command line, pynmsgpacket will open an nmsg socket output object, set it as unbuffered, and load the base:ipconn` message module:

    if args.socket:
        addr, port = args.socket.split('/')
        nmsg_out = nmsg.output.open_sock(addr, int(port))
        nmsg_out.set_buffered(False)

Instantiate nmsg input object

Next, it opens an nmsg file input object which will be used to read payloads and sets a filter so that only base:packet encoded NMSGs are returned via object read()s:

    nmsg_in = nmsg.input.open_file(args.input)
    # ensure base:packet
    nmsg_in.set_filter_msgtype('base', 'packet')

Enter payload processing loop

The program next enters the main processing loop where it reads the next payload from the input object. It stops when the payload count is reduced to 0 or the input object runs out of payloads:

    while True:
        if args.count:
            if count == 0:
                break

        msg_in = nmsg_in.read()
        if not msg_in:
            break

Emit 5-tuple to stdout

Next, if the payload contains an IPv4 TCP or UDP packet, pynmsgpacket emits the NMSG header and the packet 5-tuple to stdout:

        if msg_in['payload_type'] == "IP":
            # only process IPv4 TCP or UDP packets
            if is_ipv4_tcp_udp(msg_in):
                nmsg.print_nmsg_header(msg_in, sys.stdout)
                ip_src, port_src, ip_dst, port_dst, proto = get_pkt_info(
                                                                        msg_in)
                print "{}.{} --> {}.{} ({})".format(
                        ip_src, port_src, ip_dst, port_dst, proto)

Write base:ipconn NMSG to socket

If the user specified a remote host and port, an NMSG base:ipconn message will be constructed and written to the network:

                if args.socket:
                    msg_out = nmsg.msgtype.base.ipconn()
                    msg_out['srcip'] = ip_src
                    msg_out['dstip'] = ip_dst
                    msg_out['srcport'] = port_src
                    msg_out['dstport'] = port_dst
                    msg_out['proto'] = proto
                    t = time.time()
                    msg_out.time_sec = int(t)
                    # NMSG supports nanosecond timestamps
                    msg_out.time_nsec = int((t - int(t)) * 1E9)
                    nmsg_out.write(msg_out)

Decrement payload counter

If the user specified a counter, it gets decremented here:

        if args.count:
            count -= 1

Declare packet convenience functions

The following functions are used to winnow and extract packet details:

def is_ipv4_tcp_udp(msg):
    # AND off first 4 bits of payload (IP packet) to ensure IP version is 4
    if int(ord(msg['payload'][0])) & 0x04 == 4:
        if get_proto(msg) == socket.IPPROTO_TCP or socket.IPPROTO_UDP:
            return True
    return False


def get_pkt_info(msg):
    proto = get_proto(msg)
    ip_src, ip_dst = get_ips(msg)
    port_src, port_dst = get_ports(msg)
    return ip_src, port_src, ip_dst, port_dst, proto


def get_ips(msg):
    # IP source address is bytes 12-16, destination addres is bytes 16-20
    ip_src = socket.inet_ntoa(msg['payload'][12:16])
    ip_dst = socket.inet_ntoa(msg['payload'][16:20])
    return ip_src, ip_dst


def get_proto(msg):
    # IP protocol number is byte 9
    return ord(msg['payload'][9])


def get_ports(msg):
    # TCP/UDP port are first fields in transport header, bytes 20-22 and 22-24
    port_src = struct.unpack('!H', msg['payload'][20:22])[0]
    port_dst = struct.unpack('!H', msg['payload'][22:24])[0]
    return port_src, port_dst

Main entry point

The main body of the program is called:

if __name__ == "__main__":
    main()

Denouement

This was the last article in the Farsight Network Message tutorial series. Look for future NMSG-related articles to cover more topics including new NMSG-based tools and recipes to get the most out of the NMSG protocol.

Mike Schiffman is a Packet Esotericist for Farsight Security, Inc.