Farsight's Advanced Exchange Access Internals: Understanding Accounting

Introduction

This article explains Farsight Security’s Advanced Exchange Access (AXA)accounting subsystem. This is the mechanism by which AXA tracks, logs, andcommunicates server-side packet information.

To get the most from this article, it is recommended that you be comfortablewith the material in the following Farsight Security Blog articles:

• Farsight’s Advanced Exchange Access, Part 1
• Farsight’s Advanced Exchange Access, Part 2
• Farsight’s Advanced Exchange Access, Part 3

What is Accounting?

Accounting is AXA’s way of keeping track of traffic totals. Server-side, AXAmaintains a series of per-client packet counters (a full list is below).The AXA protocol message

AXA_P_OP_ACCT

sent from client to server is usedquery this data. The command is available from

sratool

and

radtool

as

acct

. It is also available from

sratunnel

and

radtunnel

via the

-A

command line option. Accounting messages are also logged server-side.

Accounting Glossary

The AXA accounting counters are described below.

total-collected

• : On the RAD side, packets that have been successfully run through an anomaly detector.

total-congested

• : Packets that were unable to be forwarded to the client due to connection issues, although it could also be client load (i.e., the the connection isn’t removing data from the server fast enough, or the client isn’t reading and processing the data fast enough). Note that: high server load might increase

total-missed

• , but it would generally decrease

total-congested

• by throttling the input to the congested pipe / process.

total-filtered

• : This is an SRA input counter. It measures packets/nmsgs that have been read on the input side (usually directly from the SIE network) and will be submitted to the filtering logic (i.e.: an AXA watch).

total-missed

• : This is an SRA loss counter. It measures packets that the SRA server failed to receive on the input side either because the daemon was too busy or because they were lost during transit. This counter tracks NMSG- and pcap-based loss.

total-ratelimited

• : This measures packets that were dropped instead of being forwarded to the SRA or RAD client to comply with the rate limits specified for the client.

total-sent

• : This measures packets that have been sent to an SRA or RAD client.

Accounting with sratool

Let’s have a look at a common

sratool

-based example.

sratool

is used to connect to an SRA server and a “fire hose” watch is set for our popularDNS Changeschannel.

$ sratool
sra> connect tls:mschiffm@sra-server,1021
* HELLO srad version 1.2.1 sra-server AXA protocol 1
sra> ch 214 on ; 1 wa ch=214
​[watch hits omitted]


This is left to run for approximately three minutes of wall clock time. Next,the output is paused and the accounting command is run.

sra> pause
* OK PAUSE output paused
sra> acct
* OK ACCOUNTING total-filtered=66360 total-missed=0 total-collected=0 total-sent=64912 total-ratelimited=0 total-congested=0


• We see that 66,360 nmsgs were read from the SIE channel. For thewatch we set this should actually be a good approximation of the overallpacket rate. And indeed channel 214 has an average payload rate ofapproximately 340 payloads per second. Over this very small three minute(180 second) window we observed ~369 packets.
• There were no missed packets. In this case, this is the everything’sOK alarm. We expect there to be no packet loss on the SIE input side. Ifthere were for such a low bandwidth channel, this would be indicative ofserver-side network fault or server overload.
• We see that 64,912 packets were sent from the SRA server to our

sratool

• client. This is good and what we expect.
• Because there was no congestion or rate limiting, you may wonder whythe number of sent packets is lower than the number of filtered packets.This is because when we stopped sending packets to the client, we didn’tstop the server from filtering them. SRA was still reading from the inputchannel but wasn’t forwarding any packets.

Accounting with sratunnel

Let’s have a look at another example, this time using

sratunnel

. Using thetimeout utility,

sratunnel

is invoked to run for three minutes. Itconnects to the same SRA server and sets a fire hose watch for theDNS Errorschannel. Finally, the

-A 180 -d

option string instructs

sratunnel

to emitaccounting statistics every 180 seconds and the results are written to a file.

$ timeout 180 sratunnel -s tls:mschiffm@sra-server,1021 -w "ch=220" -c 220 -A 180 -d -o nmsg:file:test-220.nmsg
connecting to tls:mschiffm@sra-server,1021
ACCOUNTING total-filtered=4183437 total-missed=44 total-collected=0 total-sent=84371 total-ratelimited=0 total-congested=4096266


• This time, 4,183,437 packets were read from the SIE channel. Using the samelogic as above, we find that this equates to approximately 23,241 payloadsper second which is commensurate to Channel 220’s searing average per secondrate of ~24,000 payloads (it is sourced from our Passive DNS feed which hasa per second payload rate of ~120,000).
• 44 packets were missed on the input side. This is a .001% rate of loss andwell within acceptable limits.
• Only 84,371 packets were sent to the client. To understand why, we look atthe congestion number.
• 4,096,266 packets were lost due to congestion. In this case, this highnumber is likely due to the fact that the client is located on the otherside of the country from the SIE data center where the SRA server islocated. Additionally, the client’s downstream Internet connection is justplain too slow to keep up with the high rate of channel 220.

Conclusion

Now that you know how to use AXA Accounting, you can use this informationalong with other data to learn more about your Farsight SRA data flows and insome cases, find and troubleshoot issues. If you’re interested in learning moreabout Farsight Products and Services, pleasereach out.

Mike Schiffman is a Packet Esotericist for Farsight Security, Inc.