Using AXAMD to Read Observations from NOD SIE Ch212 ("Newly Observed Domains") with Python3
Introduction
One of the most popular Security Information Exchange (SIE) channels is Channel 212, NOD (“Newly Observed Domains”). This channel displays domains that have never previously been seen by sensors at participating host sites. Note that NOD is not a “zone file”-based (or “CZDS”-based) solution – we’re watching for domains that are actually deployed and in use, not merely domains that have been registered.
There are many different ways to potentially access Ch212 traffic including:
- Directly via nmsgtool using a leased blade server colocated at SIE
- Using nmsgtool via an encrypted network tunnel (aka “SIE Remote Access” or “sratunnel”)
- Using SIE Batch, downloading recently seen traffic from a half day buffer of traffic
- Using incremental zone transfers (IXFR) to download Ch212 traffic in DNS RPZ format
- Using rsync to download Ch212 traffic in rbldnsdb format, and
- Via a public (but rate-limited) zone.
Today, however, we’re going to focus on using AXAMD to read Ch212 traffic from a RESTFUL API endpoint.
For reference, here’s what AXAMD stands for:
AXA = Advanced eXchange Access – the eXchange part refers to the SIE, Security Information Exchange
MD = Middleware Daemon
AXAMD = Advanced eXchange Access Middleware Daemon
AXAMD includes both a command line interface client, and a Python3 binding. We’ll show you how to use both.
Installing AXAMD
Let’s begin by creating a Python3 virtual environment where we can do our AXAMD installation:
$ /usr/local/bin/python3 -m venv ~/my-venv
$ source ~/my-venv/bin/activate
While working in that virtual environment, our normal $ prompt will be prefixed with the name of our virtual environment:
(my-venv) $ python3 -V
Python 3.10.6
If you want to routinely use that virtual environment, you may want to add the activate command to your startup dot files.
We’re now ready to clone the AXAMD repository we want to build:
(my-venv) $ git clone https://github.com/farsightsec/axamd_client.git
(my-venv) $ cd axamd_client
Installation of axamd requires just a single installation command – once all prerequisite libraries have been installed. In our case, we needed to install option_merge, jsonschema, pyyaml and pyflakes:
(my-venv) $ pip3 install option_merge
(my-venv) $ pip3 install jsonschema
(my-venv) $ pip3 install pyyaml
(my-venv) $ pip3 install pyflakes
We’re then ready to install axamd_client:
(my-venv) $ python3 setup.py installAccess to subscribed channels is controlled via API key. Use vi (or your favorite editor) to create the file~/.axamd-client.conf (note the leading tilde slash dot immediately before the rest of the filename!)
That file needs the following two lines:
server: https://axamd.sie-remote.net
apikey: put-your-actual-SIE-Remote-Access-API-key-hereNote that your SIE Remote Access API key is NOT the same as the DNSDB API key that you may also have!
After saving that file, make sure you’re the only one able to read it:
(my-venv) $ chmod go-rwx ~/.axamd-client.confRunning the AXAMD Unit Tests
You’re now ready to run the provided unit tests.
If you’re running a current version of Python3, you’ll probably need to make one small tweak. Before we could successfully run the full set of AXAMD unit tests, ~/my-venv/lib/python3.10/site-packages/option_merge/merge.py needed to be updated so that
from collections import Mappingbecomes:
from collections.abc import MappingHaving done that, we could then successfully run the full set of unit tests from the ~/axamd_client directory:
(my-venv) $ python3 setup.py test
[bunch of output elided here]
----------------------------------------------------------------------
Ran 64 tests in 0.245s
OK
Trying the axamd_client Command Line Interface (CLI) client
Part of what you’ve now got installed is the axamd_client Command Line Interface client. This may be all some people need for their AXAMD-related work. You can see a command line summary by entering:
(my-venv) $ axamd_client -hNow let’s try using axamd_client to confirm we have access to Channel 212:
(my-venv) $ axamd_client --list-channels
[...]
ch212:
10.32.212.255/8430 10.32.212.255/9430
[...]
If you DON’T see ch212 listed, contact your DomainTools Account Executive to ensure you’ve subscribed to that channel.
Now let’s actually grab some SIE NOD traffic!
If we don’t want axamd_client to “run forever,” we can ask to collect either a specific number of observations (for example, --number 1000) or for a specific period of time (for example, --duration 15s).
Let’s try collecting 15 seconds worth of NOD observations, writing them to the temporary file temp.jsonl
(my-venv) $ axamd_client --duration 15s --channel 212 --watches ch=212 > temp.jsonlIn our test run, we got 40 observations in just that 15 second period; your test results may end up being larger or smaller:
(my-venv) $ wc -l temp.jsonl
40 temp.jsonl
Helpful hint: when specifying time durations, you can conveniently use either a hh:mm:ss or #w#d#h#m#s format.
Looking at Some of Our Results with jq
Having collected some data we’re now ready to scrutinize it. That output is in JSON Lines format (https://jsonlines.org/) and can be read pretty easily with the “naked eye,” but we’ll still use jq (https://stedolan.github.io/jq/) to “pretty print” our observations:
(my-venv) $ jq '.' < temp.jsonl | more
{
"tag": 1,
"op": "WATCH HIT",
"channel": "ch212",
"vname": "SIE",
"mname": "newdomain",
"time": "2022-08-02 19:35:04.643621570",
"nmsg": {
"time": "2022-08-02 19:35:04.643621570",
"vname": "SIE",
"mname": "newdomain",
"source": "a1ba02cf",
"message": {
"domain": "sellmypennsylvaniahomefast.com.",
"time_seen": "2022-08-02 19:34:45",
"bailiwick": "com.",
"rrname": "sellmypennsylvaniahomefast.com.",
"rrclass": "IN",
"rrtype": "NS",
"rdata": [
"dns1.registrar-servers.com.",
"dns2.registrar-servers.com."
],
"keys": [],
"new_rr": []
}
}
}
[etc]
Note that newly observed entries seen in NOD may range from terrific to malicious. We do NOT “defang” the output you receive from AXAMD, so exercise due care when investigating any newly observed domains you see reported – many will be fine, but some may be risky to investigate.
While we used jq just to improve the readability of our JSON Lines output, it can do far more. For example, if you just want to get a raw list of domains from your output, jq can help with that:
(my-venv) $ jq -r '.nmsg.message.domain' < temp.jsonl“What Are These “ACCOUNTING” Messages I Periodically See in My Output?”
Most of your AXAMD output will normally be "WATCH HIT" entries like the example shown above, but periodically you may also see "ACCOUNTING" reporting summary statistics. A sample looks like:
{"tag":"*","op":"OK","orig_op":"ACCOUNTING","str":"total-filtered=29 total-missed=0 total-collected=0 total-sent=29 total-ratelimited=0 total-congested=0"}AXAMD will easily keep up with comparatively low bandwidth SIE channels such as Ch212, so these accounting messages won’t be particularly exciting or informative. You can control the interval between ACCOUNTING entries being emitted with the axamd_client --report-interval option (including setting that interval to zero to turn them off altogether).
“But I Want to Integrate AXAMD Directly into My Own Python Code! How Do I Do THAT?”
Here’s some sample code, with the tiny chunk of AXAMD-specific code rendered in bold. Those couple of AXAMD calls are pretty straightforward, with the rest of the code just being a matter of handling required imports, a few lines of “boiler plate” code to handle CTRL-C interrupts, and a couple of regular expressions to pick up the API key and server name from the config file.
#!/usr/bin/env python3 # pylint: disable=C0114
import sys
import re
import json
import signal
from pathlib import Path
from axamd.client import Client # pylint: disable=E0401
# handle CTRL-C cleanly
def handler(mysig, myframe): # pylint: disable=W0613,C0116
sys.exit('')
signal.signal(signal.SIGINT, handler)
# get the SIE Remote Access API key and AXAMD server
my_cfg_file = str(Path.home()) + "/.axamd-client.conf"
with open(my_cfg_file, 'r', encoding='utf8') as mycfg:
contents = mycfg.read()
mypat1 = re.compile('apikey: (.*)')
apikey = mypat1.findall(contents)[0]
mypat2 = re.compile('server: (.*)')
server = mypat2.findall(contents)[0]
# Connect to the assigned SIE Remote Access server with their API key
c = Client(server, apikey)
# read Ch212 and print out the JSON Lines format results
for line in c.sra(channels=[212], watches=['ch=212']):
data = json.loads(line)
print(data)Assuming that code is in a file called sample_run.py, a sample run looks like:
(my-venv) $ ./sample_run.py
{'tag': 1, 'op': 'WATCH HIT', 'channel': 'ch212', 'vname': 'SIE', 'mname': 'newdomain', 'time': '2022-08-03 00:49:46.182181816', 'nmsg': {'time': '2022-08-03 00:49:46.182181816', 'vname': 'SIE', 'mname': 'newdomain', 'source': 'a1ba02cf', 'message': {'domain': 'liftedandloweredgolfcarts.com.', 'time_seen': '2022-08-03 00:48:09', 'bailiwick': 'com.', 'rrname': 'liftedandloweredgolfcarts.com.', 'rrclass': 'IN', 'rrtype': 'NS', 'rdata': ['ns10.wixdns.net.', 'ns11.wixdns.net.'], 'keys': [], 'new_rr': []}}}
[etc]
[hit CTRL-C to kill the streaming session]
Conclusion
We hope you’ve enjoyed learning how to get, build and use AXAMD to collect data from NOD (SIE Channel 212).
If you’d like to talk with DomainTools in order to subscribe to NOD and AXAMD, please contact us.
