Abstract
This article is the fifth and final in a multi-part blog series intended tointroduce and acquaint the user with Farsight Security’s NMSG suite. Thisarticle introduces the
pynmsg
Python programming API.
Before reading this article, it is recommended that you read the followingFarsight Security Blog articles:
- Farsight’s Network Message, Volume 1: Introduction to NMSG
- Farsight’s Network Message, Volume 2: Introduction to nmsgtool
- Farsight’s Network Message, Volume 3: Headers and Encoding
- Farsight’s Network Message, Volume 4: The C Programming API
This article, geared towards intermediate-level Python programmers, is by nomeans an exhaustive API reference. This article covers NMSG (protocol)version
2
and
pynmsg
version
0.2
.
The reader is also directed to explore the examples directory in the
pynmsg
repository, whereseveral other sample
pynmsg
-based programs will be found.
Python and Cython
pynmsg
is a Python extension moduleimplemented in Cython for the nmsg C library. This article shows the reader how to instantiate an
nmsg
session using Python and then read and write NMSGs. It assumes youalready have
nmsg
and
pynmsg
installed.
pynmsgpacket
pynmsgpacket
is a command-line tool that processes
base
:
packet
encodedNMSGs. It winnows payloads down to those containing only TCP and UDP packetsand then extracts a 5-tuple (source IP, source port, destination IP,destination address, protocol) and prints it to stdout. If so configured, theprogram will then use this 5-tuple to construct a new
base
:
ipconn
NMSGmessage and forward to a network listener.
Loyal readers will recognize
pynmsgpacket
builds on lessons learned fromthe previous NMSG blog article and sources input from files created by nmsgpacket.
First, let’s demonstrate the finished product:
$ ./pynmsgpacket
usage: pynmsgpacket [-h] [-c COUNT] [-s SOCKET] [input]
Print IPv4 TCP or UDP packet data from base:packet encoded NMSGs and
optionally write base:ipconn NMSGs to a remote listener
positional arguments:
input input file, also accepts input from pipeline
optional arguments:
-h, --help show this help message and exit
-c COUNT, --count COUNT
stop after count payloads
-s SOCKET, --socket SOCKET
write binary NMSGs to socket (i.e., 127.0.0.1/9430)
We can process a few payloads from a previously created
nmsgpacket.nmsg
file:
$ pynmsgpacket -c 3 nmsgpacket.nmsg
[2015-02-22 06:58:27.573210000] [1:12 base packet] [] [] []
10.0.1.52.17500 --> 255.255.255.255.17500 (17)
[2015-02-22 06:58:27.573489000] [1:12 base packet] [] [] []
10.0.1.52.17500 --> 10.0.1.255.17500 (17)
[2015-02-22 06:58:30.32007000] [1:12 base packet] [] [] []
172.16.82.195.80 --> 10.0.1.51.59451 (6)
More complex command-line invocations are available as well. First, instantiatean
nmsgtool
listener:
$ nmsgtool -l 10.0.1.52/9430
Then invoke
nmsgtool
to read from the network, encode as
base
:
packet
andpipe the NMSGs to
pynmsgpacket
which will dump the 5-tuple to stdout and thenemit
base
:
ipconn
NMSGs to the remote listener:
$ nmsgtool -i en0 -V base -T packet -w - --unbuffered | pynmsgacket -s 10.0.1.52/9430
[2015-02-22 20:54:06.253410000] [1:12 base packet] [00000000] [] []
10.0.1.51.61929 --> 10.0.1.52.22 (6)
[2015-02-22 20:54:06.253410000] [1:12 base packet] [00000000] [] []
10.0.1.51.61929 --> 10.0.1.52.22 (6)
[2015-02-22 20:54:06.253410000] [1:12 base packet] [00000000] [] []
10.0.1.51.61929 --> 10.0.1.52.22 (6)
[2015-02-22 20:54:06.253418000] [1:12 base packet] [00000000] [] []
10.0.1.51.61929 --> 10.0.1.52.22 (6)
...
At the console running the first instantiation of
nmsgtool
, the following isemitted:
[20] [2015-02-22 20:54:07.143420934] [1:5 base ipconn] [00000000] [] []
proto: 6
srcip: 10.0.1.51
srcport: 61929
dstip: 10.0.1.52
dstport: 22
[20] [2015-02-22 20:54:07.143464088] [1:5 base ipconn] [00000000] [] []
proto: 6
srcip: 10.0.1.51
srcport: 61929
dstip: 10.0.1.52
dstport: 22
[20] [2015-02-22 20:54:07.143507957] [1:5 base ipconn] [00000000] [] []
proto: 6
srcip: 10.0.1.51
srcport: 61929
dstip: 10.0.1.52
dstport: 22
[20] [2015-02-22 20:54:07.143552064] [1:5 base ipconn] [00000000] [] []
proto: 6
srcip: 10.0.1.51
srcport: 61929
dstip: 10.0.1.52
dstport: 22
pynmsgpacket source
The following is the entire 129 line Python source code file with in-lineannotations explaining
pynmsg
API calls. As a matter of note,
pynmsgpacket
is written as a tutorial and as such doesn’t have a lot of things proper Pythoncode should, like docstrings and try/except clauses wrapping function callsthat can raise exceptions.
The full source code is available for download from Farsight Security’s blog-code GitHub page.
Preamble
The first section contains the source code license and the standard Pythonimports:
#!/usr/bin/env python
# Copyright (c) 2015 by Farsight Security, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# pynmsgpacket
import sys
import time
import struct
import socket
import argparse
import nmsg
Handle command-line arguments
Upon entering the
main()
function body,
pynmsgpacket
parses command-linearguments. The main thing it figures out here is whether or not to look forinput NMSGs from a file or from a pipe. It also sets the count:
def main():
parser = argparse.ArgumentParser(
description="Print IPv4 TCP or UDP packet data from base:packet"
" encoded NMSGs and optionally write base:ipconn NMSGs to a"
" remote listener")
parser.add_argument('input', nargs='?', type=argparse.FileType('r'),
help="input file, also accepts input from pipeline")
parser.add_argument("-c", "--count", type=int,
help="stop after count payloads")
parser.add_argument("-s", "--socket",
help="write binary NMSGs to socket"
" (i.e., 127.0.0.1/9430)")
args = parser.parse_args()
if not sys.stdin.isatty():
args.input = sys.stdin
if not args.input or args.count == 0:
parser.print_help()
exit(1)
count = args.count or 0
Instantiate nmsg output object
If the user specifies
-s
at the command line,
pynmsgpacket
will open an
nmsg
socket output object, set it as unbuffered, and load the
base:
ipconn`message module:
if args.socket:
addr, port = args.socket.split('/')
nmsg_out = nmsg.output.open_sock(addr, int(port))
nmsg_out.set_buffered(False)
Instantiate nmsg input object
Next, it opens an
nmsg
file input object which will be used to read payloadsand sets a filter so that only
base
:
packet
encoded NMSGs are returned viaobject
read()
s:
nmsg_in = nmsg.input.open_file(args.input)
# ensure base:packet
nmsg_in.set_filter_msgtype('base', 'packet')
Enter payload processing loop
The program next enters the main processing loop where it reads the nextpayload from the input object. It stops when the payload count is reduced to 0or the input object runs out of payloads:
while True:
if args.count:
if count == 0:
break
msg_in = nmsg_in.read()
if not msg_in:
break
Emit 5-tuple to stdout
Next, if the payload contains an IPv4 TCP or UDP packet,
pynmsgpacket
emitsthe NMSG header and the packet 5-tuple to stdout:
if msg_in['payload_type'] == "IP":
# only process IPv4 TCP or UDP packets
if is_ipv4_tcp_udp(msg_in):
nmsg.print_nmsg_header(msg_in, sys.stdout)
ip_src, port_src, ip_dst, port_dst, proto = get_pkt_info(
msg_in)
print "{}.{} --> {}.{} ({})".format(
ip_src, port_src, ip_dst, port_dst, proto)
Write base:ipconn NMSG to socket
If the user specified a remote host and port, an NMSG
base
:
ipconn
messagewill be constructed and written to the network:
if args.socket:
msg_out = nmsg.msgtype.base.ipconn()
msg_out['srcip'] = ip_src
msg_out['dstip'] = ip_dst
msg_out['srcport'] = port_src
msg_out['dstport'] = port_dst
msg_out['proto'] = proto
t = time.time()
msg_out.time_sec = int(t)
# NMSG supports nanosecond timestamps
msg_out.time_nsec = int((t - int(t)) * 1E9)
nmsg_out.write(msg_out)
Decrement payload counter
If the user specified a counter, it gets decremented here:
if args.count:
count -= 1
Declare packet convenience functions
The following functions are used to winnow and extract packet details:
def is_ipv4_tcp_udp(msg):
# AND off first 4 bits of payload (IP packet) to ensure IP version is 4
if int(ord(msg['payload'][0])) & 0x04 == 4:
if get_proto(msg) == socket.IPPROTO_TCP or socket.IPPROTO_UDP:
return True
return False
def get_pkt_info(msg):
proto = get_proto(msg)
ip_src, ip_dst = get_ips(msg)
port_src, port_dst = get_ports(msg)
return ip_src, port_src, ip_dst, port_dst, proto
def get_ips(msg):
# IP source address is bytes 12-16, destination addres is bytes 16-20
ip_src = socket.inet_ntoa(msg['payload'][12:16])
ip_dst = socket.inet_ntoa(msg['payload'][16:20])
return ip_src, ip_dst
def get_proto(msg):
# IP protocol number is byte 9
return ord(msg['payload'][9])
def get_ports(msg):
# TCP/UDP port are first fields in transport header, bytes 20-22 and 22-24
port_src = struct.unpack('!H', msg['payload'][20:22])[0]
port_dst = struct.unpack('!H', msg['payload'][22:24])[0]
return port_src, port_dst
Main entry point
The main body of the program is called:
if __name__ == "__main__":
main()
Denouement
This was the last article in the Farsight Network Message tutorial series.Look for future NMSG-related articles to cover more topics including newNMSG-based tools and recipes to get the most out of the NMSG protocol.
Mike Schiffman is a Packet Esotericist for Farsight Security, Inc.
Heading
