Skip to main content
DomainTools Announces Predictive Threat Feeds - Powering Preemptive Exposure Management
Tag title

Advanced Exchange Access (AXA) Toolkit User Guide

Published on: 
December 9, 2022
On This Page
TOC Element
Share:

Introduction

Farsight Security® Inc.'s (now a part of DomainTools) Security Information Exchange (SIE) is the world's largest real-time threat intelligence platform — it aggregates, filters and broadcasts diverse Internet-security related information so security professionals can more accurately and quickly identify, map, and protect from cybercriminal activity.

There are multiple delivery mechanisms to consume data on SIE:

  • SIE Remote Access
    • Farsight Advanced Exchange Access Toolkit
  • Farsight Advanced Exchange Access RESTful Interface
  • SIE Direct Access
    • SIE Blade
    • SIE Port Access
      • Cross-connect
      • Server Co-location

This user guide will discuss and illustrate the tools found in theFarsight Advanced Exchange Access Toolkit to connect and consume datafrom SIE.

Advanced Exchange Access Toolkit

The Advanced Exchange Access (AXA) toolkit contains tools and a Clibrary to bring Farsight's real-time data and services directly fromthe Farsight Security Information Exchange (SIE) to the subscriber'snetwork edge.

AXA enables subscribers to connect to Farsight's subscription-based SRA(SIE Remote Access) and RAD (Real-time Anomaly Detector) servers. Theseservers provide access to data and services built from Farsight's SIE.

SRA streams real-time SIE data while RAD streams real-time anomalydetection data (from services such as Brand Sentry and Domain Sentry).

Audience

This document is intended for system administrators and programmers whowant to interact with SIE.

Requirements

Operating System

Linux, FreeBSD or other POSIX compliant operating systems.

Hardware

The minimum hardware requirements to get started with tools in theAdvanced Exchange Access Toolkit are listed below. Depending on theamount of data being processed, the resources may need to be increasedaccordingly.

  • 1 CPU
  • 1 GB Memory
  • 1 GB Disk

Network

Tools in the Advanced Exchange Access Toolkit require permitted outboundto sra.sie-remote.net and rad.sie-remote.net over TCP using port 22.

Service Entitlement

Subscribers must have purchased a SIE service entitlement from FarsightSecurity and have been provisioned access using a SSH key.

Contents of the Advanced Exchange Access Toolkit

The Advanced Exchange Access Toolkit distribution contains the following:

  • sratool: A test/debug/instructional command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data.
  • radtool: A test/debug/instructional command-line tool used to connect to a RAD server, set watches, enable anomaly detection modules, and stream data.
  • sratunnel: A production command-line tool that streams SIE data to the local network.
  • radtunnel: A production command-line tool that streams anomaly data to the local network.
  • libaxa: A C library providing an API for the AXA protocol including:
    • connection instantiation/teardown,
    • message encapsulation/decapsulation,
    • watch parsing/loading,
    • trie storage and lookup,
    • control packet rate limits, sampling rates, window sizes, and many other AXA-specific functions.

For usage details on sratool, radtool, sratunnel, and radtunnel,please see their respective man pages (included in the distribution).

Installing Advanced Exchange Access Toolkit (axa-tools)

Debian 8 and Ubuntu 14.04/16.04

These instructions use Debian packages created, maintained and hosted byFarsight Security.

  • Download the Farsight Apt signing key.

$ sudo wget -O /etc/apt/trusted.gpg.d/debian-farsightsec.gpg https://dl.farsightsecurity.com/debian/archive.pubkey

  • Add the Farsight Debian repository.

$ echo "deb http://dl.farsightsecurity.com/debian wheezy-farsightsec main" | sudo tee -a /etc/apt/sources.list.d/debian-farsightsec.list

  • Resynchronize the package index files.

$ sudo apt update

  • Install the Advanced Exchange Access Toolkit (axa-tools).

$ sudo apt install axa-tools

Build from Source

See the section titled Building manually in the READMEfile found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit

Configuring Advanced Exchange Access with SSH

  • At the time of provisioning you would have been asked to generate a SSH key pair used for authentication. The following steps will reference this key, make sure you reference the correct directory path when configuring the key.

$ ssh-keygen -t rsa -b 4096 -C farsight_security -f ~/.ssh/farsight_security

  • Create or edit the SSH config file with the following:

$ vim ~/.ssh/config

Add the following:

Host sra.sie-remote.net rad.sie-remote.net
IdentityFile ~/.ssh/farsight_security

Prerequisites

  • A host with Linux, FreeBSD or other POSIX compliant operating system installed
  • A SIE Remote Access entitlement from Farsight Security
  • Having already exchanged a SSH keypair with Farsight Security
  • Having installed and configured the Advanced Exchange Access Toolkit

Usage Examples

sratool

sratool is a test/debug/instructional command-line tool used toconnect to an SRA server, set watches, enable SIE channels, and streamdata.

Stream SIE traffic with sratool

An example using sratool to emit five messages seen on SIE Channel 255(SIE Heartbeat Channel):

  1. $ sratool
  2. sra> connect ssh:sra-service@sra.sie-remote.net: connect to an SRA server using the SSH transport. SSH used its keyring to prove the user's identity, so there was no 'password:' prompt. The HELLO response from the remote end displays its version number and the protocol level.
  3. sra> count 5: instruct the sratool client to stop after five messages are output.
  4. sra> channel 255 on: instruct the remote end to listen to SIE channel 255 which was OK'd by the server indicating that it is provisioned for this channel according to the authentication and authorization level.
  5. sra> 1 watch ch=255: watch all content on channel 255 (with no rate limiting or filtering).

$ sratool
sra> connect ssh:sra-service@sra.sie-remote.net
HELLO srad version 1.2.1 sra AXA protocol 1
sra> count 5
sra> channel 255 on
OK CHANNEL ON/OFF channel ch255 on
sra> 1 watch ch=255
1 OK WATCH started
1 ch255 base encode TEXT
1 ch255 base encode TEXT
1 ch255 base encode TEXT
1 ch255 base encode TEXT
1 ch255 base encode TEXT
packet count limit exceeded
sra> exit

sratunnel

sratunnel is a production command-line tool that streams SIE data to the local network.

Create a persistent connection to SIE

An example using sratunnel as a background process to stream nmsgmessages from SIE Channel 255 (SIE Heartbeat Channel) to theloopback interface on port 8000.

  1. Invoke sratunnel with the following arguments.

$ sratunnel -s 'ssh:sra-service@sra.sie-remote.net' -c 255 \
-w ch=255 -o nmsg:udp:127.0.0.1,8000 &

  1. Use tcpdump to confirm messages are being streamed.

$ sudo tcpdump -i lo -c 5 -nn port 8000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
11:18:41.204425 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:18:58.672776 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:19:16.312962 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:19:33.833821 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:19:51.277784 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
5 packets captured
10 packets received by filter
0 packets dropped by kernel

  • Bring the background process to the foreground.

$ fg

  • Kill the sratunnel process by pressing Control-C.

Process messages with nsmgtool

The nmsgtool program is a single tool for taking inputs from a varietyof different inputs like data streams from the network, capturing datafrom network interfaces, reading data from files or even standard inputand making NMSG payloads available to one or more outputs.

Installing nmsgtool

  • If you have not already installed nmsgtool, install it:

$ sudo apt install nmsgtool nmsg-msg-module-sie

Viewing a stream of nmsg's from sratunnel with nmsgtool

An example using sratunnel as a background process to stream nmsgmessages from SIE Channel 255 (SIE Heartbeat Channel) to the loopbackinterface on port 8000; using nmsgtool to connect to the loopbackinterface and print the nmsg to the terminal in presentation format.

  • Invoke sratunnel with the following arguments:

$ sratunnel -s 'ssh:sra-service@sra.sie-remote.net' -c 255 -w ch=255 -o nmsg:udp:127.0.0.1,8000 &

  • Invoke nmsgtool to connect to the loopback interface on port 8000, process three payloads and print the output to the terminal using the presentation format.

$ nmsgtool -l 127.0.0.1/8000 -c 3 -o -
[23] [2017-06-28 19:53:51.844574928] [1:11 base encode] [1ba02cfd] [] []
type: TEXT
payload:

[23] [2017-06-28 19:53:52.345241069] [1:11 base encode] [1ba02cfd] [] []
type: TEXT
payload:

[23] [2017-06-28 19:53:52.845875978] [1:11 base encode] [1ba02cfd] [] []
type: TEXT
payload:

  • Bring the background process to the foreground.

$ fg

  • Kill the sratunnel process by pressing Control-C.

Saving a stream of nmsg's from sratunnel with nmsgtool

An example using sratunnel as a background process to stream nmsgmessages from SIE Channel 255 (SIE Heartbeat Channel) to the loopbackinterface on port 8000; using nmsgtool to connect to the loopbackinterface and saving the output to a rotating set of files using thenmsgtool kicker function.

  • Invoke sratunnel with the following arguments:

$ sratunnel -s 'ssh:sra-service@sra.sie-remote.net' -c 255 -w ch=255 -o nmsg:udp:127.0.0.1,8000 &

  • Invoke nmsgtool to connect to the loopback interface on port 8000, save nmsg files to disk every sixty seconds as a background process.

$ nmsgtool -l 127.0.0.1/8000 -t 60 -k '/bin/true' -w ch255 &

  • List the saved files using ls.

$ ls -l
total 16
-rw-r--r-- 1 demo demo 5518 Jun 28 16:03
ch255.20170628.2002.1498698127.548592412.nmsg
-rw-r--r-- 1 demo demo 6436 Jun 28 16:04
ch255.20170628.2003.1498698180.574404303.nmsg

  • Read one of the files using nmsgtool and outputting the results to the terminal in JSON:

$ nmsgtool -r ch255.20170628.2003.1498698180.574404303.nmsg -J -
{"time":"2017-06-28 20:03:02.061745882","vname":"base",
"mname":"encode","source":"1ba02cfd",
"message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}}
{"time":"2017-06-28 20:03:02.562045097","vname":"base",
"mname":"encode","source":"1ba02cfd",
"message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}}
{"time":"2017-06-28 20:03:03.062705039","vname":"base",
"mname":"encode","source":"1ba02cfd",
"message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}}

  • Bring the nmsgtool background process to the foreground.

$ fg

  • Kill the nmsgtool process by pressing Control-C.
  • Bring the sratunnel background process to the foreground.

$ fg

  • Kill the sratunnel process by pressing Control-C.

AXA Protocol

The AXA protocol is documented in the section titled AXA Protocol in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit.

Limits

Some of the channels offered by the SIE network burst to an extremelyhigh bitrate (some over 500Mbps). AXA has two ways to deal with suchnetwork-hungry situations: optional filtering and loss-tolerance builtinto the protocol.

Filtering can take one of the following forms:

  • Via the rate limit option to reduce the flow of ingress data to a certain number of packets per second.
  • Via one or more IP-based or DNS-based "watches" to limit the flow of data to specific assets the subscriber wishes to observe.

Finally, AXA is a deliberately lossy protocol. If a subscriber requests more data than the network can carry, data overruns will occur. When this happens, loss markers are transmitted reliably within the AXA stream to inform the subscriber via the AXA accounting subsystem. At this point, the subscriber’s possible mitigation strategies include:

  • ask for less data via rate limiting
  • increase network capacity and/or other host resources
  • treat the SRA stream as a chunky and non-representative sample of the total SIE data
  • pursue Direct Access to SIE

Additional Information