Introduction
You may sometimes wonder what prompts people to get started in the crazy worldof system and network security work, or how they got started fighting cybercrime and network abuse. Why would anyone fight hacker/crackers, or phishingand online fraud, or spamming?
For a surprising number of older anti-abuse people, the answer may be“Usenet.”Usenet began way back in 1980, and is a service that easily predates the WorldWide Web.
Usenet was, and remains, a sort of distributed international bulletin boardsystem. Usenet messages look a lot like email messages, except instead of beingsent person-to-person, users “post” or “submit” messages to one or more Usenetnewsgroups. If you wanted to read articles about a particular computer sciencetopic, talk a little politics, discuss chess, or chat about what was going onin France or Spain or Germany (in French or Spanish or German no less), therewas likely a newsgroup meant just for that purpose. Usenet news also carried alot of text-encoded binary content, particularly in the
alt.*
(alternative)newsgroup hierarchy.
The administrator of a Usenet server at each site arranged to “peer” with newsservers at other sites, offering new articles from the server’s spool via aflooding protocol known as NNTP (Network News Transfer Protocol).
Because new articles were constantly coming in, and spool storage space isfinite (or at least it was back then), old articles would be periodicallyexpired (removed) to make room for new ones. Duplicates were prevented throughthe use of unique message-IDs, loops were prevented with path headers updatedon a hop-by-hop basis, and transfers were optimized with a variety oftechniques including parallel message streams and cyclical news file systems.All-in-all, Usenet was a pretty cool distributed information sharingenvironment, and one that still exists today. In fact, as of the start of thisyear, Usenet traffic volume had increased to the point where it was runningover 17 TB per day*.
If you find yourself intrigued by the thought that there’s a part of theInternet you didn’t know existed, and you’d like to check it out, a list of3rd party Usenet providers can be seenhere.
Bottom line, if you’re a user, Usenet was (and is) a terrific place to discussvarious topics. If you were a sysadmin responsible for running a Usenet newsserver, it was (and is) a terrific proving ground for high performancenetworking, disk-intensive I/O prototyping, and so on.
It also, unfortunately, tended to quickly motivate a personal and professionalinterest in anti-abuse efforts.
Usenet Abuse
The continued usability of Usenet has always really depended on the courtesy ofits users. If a Usenet newsgroup was devoted to high energy physics, commonsense dictates that articles about scuba diving or recent movies would beoff-topic. Those discussions really should be held in a more appropriatenewsgroup, instead.
Most users respected that, and were careful to post to appropriate newsgroups.Some, however, simply posted whatever they wanted, wherever they wanted, eitherbecause they were confused or careless, or had malicious intent, or simplywanted to blast their commercial message to everyone since the cost to them ofdoing so was effectively zero. These behaviors, if tolerated, could obviouslycause problems for users and Usenet administrators alike: discussions would getderailed, users would complain to administrators, spool space might becomeexhausted prematurely, etc., etc., etc.
Kill files, implemented client-side in News reading client software, allowedusers to selectively address minor irritations by automatically and silentlysuppressing the display of posts made by certain obnoxious people.
Larger-scale and more serious Usenet abuse issues were often handled withcancel messages. Cancel messages were originally envisioned as a way for anindividual user to request deletion of one of their own posts (e.g., if youdeveloped second thoughts about a message you might have posted accidentally orin the heat of the moment, you could send a cancel message and it would haveyour earlier posting deleted at many sites). However, because cancel messagesweren’t cryptographically authenticated, cancel messages could be “forged” todelete ANY arbitrary message, regardless of who originated it, at least if thesite receiving the cancel messages chose to honor/process those cancelmessages.
This lead to some of the first “bot wars:” automatic spam bots (run by the badguys) would post waves of garbage to Usenet newsgroups, while automatic cancelbots (run by the good guys) would rapidly clean up that mess by issuing cancelsfor those messages as quickly as they were posted.
Of course, the bad guys then tried to discourage sites from accepting cancelsat all by issuing forged cancels for ALL articles in Usenet… and the goodguys responded by “aliasing out” (systematically shunning) all traffic from theproblematic servers that were trying to “cancel everything,” a classic “armsrace.”
Spam wasn’t the only sort of problematic content in Usenet. Fraudulent scams ofvarious sorts, for instance, were seen from time to time in Usenet just asthey’re seen today in email, social media, and elsewhere.
And because text-encoded binaries were a material part of a typical feed, thepotential for Usenet to act as a vector for malware distribution was also veryreal. Binary content also meant that copyright infringing content (such aspirated software, pirated music, and pirated movies) was another potentiallyproblematic area, as was illegal online child sexual abuse material, althoughwhat was present in a given server’s spool would vary dramatically depending onwhat newsgroups your News admin elected to carry.
In spite of all these real or potential issues, Usenet resulted in a very richand creative intellectual environment that facilitated a lot of productivework. It was a true community, and at least among some Usenet administrators,the impetus for an intense focus on system and network performance, andmeasurement work.
By way of example, one colleague at Unidata in Colorado evaluated using NNTP as a potential data distribution protocol for pushing binary weather data. She found that from a latency andarticle completeness perspective, NNTP rocked.
The peer-to-peer nature of the Usenet feed environment also resulted in manyUsenet system administrators forming close collaborative relationships witheach other. Relationships of that sort often formed the foundation for laterabuse mitigation efforts, and were as important as technical advances.
How Does This Relate to Cyber Security Today, and Farsight?
Farsight is a cyber security data company. Multiple Farsight staff members haveat least historical connections to Usenet, AND deep connections to theanti-abuse and security communities. Now, having read along to this point, youknow a little bit about why those connections tend to exist.
You now also understand a little about why we care about data distributiontechnologies: whether folks were pushing Usenet articles in the old days orpushing cyber security data today, we want and need to quickly, efficiently andscalably move large quantities of data over the wire. This means we care aboutadvanced networking, I/O optimization, data structures, data transferprotocols, securing data in flight and at rest, traffic measurement andanalysis, etc.
We also count on our colleagues and friends to help us continue to win thefight against cyber crime and other online abuse. Just like personally-arrangedUsenet newsfeeds, collaborative data-driven security only works if peopleagree to share. It’s your data, your contribution, your telemetry that youshare with Farsight, that makes all the difference. We deeply appreciate yourgenerosity and we literally couldn’t continue the fight against the bad guyswithout your help and your data. A big THANK YOU to all our sensor operatorsout there!
We’d also like to take this opportunity to remind researchers at accreditedacademic institutions that we welcome the chance to help support your workwith full or partially underwritten grant access to Farsight Security’s data.
Looking Forward
We’re also curious: while we don’t currently work with Usenet data as a cybersecurity data source, is there data actually in Usenet traffic that you thinkwould be helpful to cyber security and anti-abuse efforts? Is this an areawhere you’re currently lacking visibility, and need to fix that? After all,there’s probably something going on in over 17TB of data a day — wouldn’t itbe nice to know what?
Anyhow, we always like to hear from our friends and customers. Please feel freeto send along feedback.
* 17TB/day ==> 17,000,000 MB * 8 bits per byte / (24 hrs/day * 60 min/hr * 60seconds per minute==> ~1.574 Gigabits/second, assuming traffic is uniform(which it isn’t) and there’s no duplication of incoming traffic (which therewill be). See here for more information.
Joe St Sauver, Ph.D. is a Scientist with Farsight Security, Inc.
Heading
