Being an Internet Packrat

Introduction

Data passes across the Farsight Security, Inc.Security Information Exchangeevery second of every day. While a significant percentage of it is related toPassive DNS replication, we occasionally participate in sinkhole administrationefforts for botnet cleanup operations. We forward that information in realtime to participating organizations that in turn fan that data out to ISPs andInternet security companies to help identify and remediate affected clients.

As part of curating that feed, and with the permission of the sinkholeoperator, we also normally archive that data. Doing so has paid off forapproved and vetted researchers as we can make that data available forsubsequent retrospective analysis.

Some examples follow.

“Post-Mortem of a Zombie: Conficker Cleanup After Six Years”

The Conficker sinkhole is the longest effort we’ve facilitated. After theConficker botnet wassinkholed, infected Windows clientshave continued to try to “phone home” for updates for almost seven years.Today, the Conficker botnet may be about a sixth of the size it was at itspeak, but it’s still present, on up to one million clients.

We’ve kept a Conficker data archive online in our data centers for download byresearchers and remediators alike, but it wasn’t until recently that a teamfrom Technical University Delft (NL) completed a study of long-term remediationefforts utilizing that data. Their work was accepted and presented at UsenixSecurity 2015 earlier this month (see link above).

”DNS Changer Remediation Study”

When we took part in the takedown of the DNS Changer botnet, we made periodic archives available for download bydifferent remediation teams, and we maintained the data for as long as wecould. A group from Georgia Tech (US) was able to take the data and use it tocompare different notification methods utilized during remediation efforts todetermine which were more effective.

“Developing Security Reputation Metrics for Hosting Providers”

More recently, DNS researchers at TU Delft (NL) wanted to look back, on a dailybasis, at Passive DNS DNSDB data, something we don’t normally keep (it getsrolled up into monthly databases). By special arrangement, we kept daily DNSDBdatabase dumps going long enough for them to utilize correlations from DNSresources to develop security metrics for ISPs and hosting providers. Theirwork was also just recently published earlier this month at Usenix Security(CSET ’15).

Lessons Learned

Along the way we’ve learned many lessons about long-term storage that might behelpful for other projects:

• • ). A tried and true method for storing all manner of data, when used to store sinkhole data this makes it easier to manage migrating storage sets between systems or storage chassis. A client can access multiple file servers via NFS, and the directory structure provides a clear boundary for NFS mount points. If users are downloading data regularly, splitting data into multiple directories also helps programs like rsync limit how many files need to be processed during synchronization, reducing transfer overhead.
  • Where possible, always log data twice. Log once on the sinkhole data collection system and once where the data is aggregated and permanently archived. The sinkhole storage can be automatically purged after a few days, but having it around for a few days provides the ability to recover from a temporary logging or storage problem in the long-term archive storage. Monitoring data availability as if you were a client will also help catch issues in time for you to recover utilizing data from the limited storage on the sinkhole server.
  • Be clear about your data retention policies. If there is a lack of an understanding between all parties at the start of a collection effort, many might assume that we can collect data indefinitely and not ask us to stop. Several times we’ve had to look for more storage resources for Conficker data, and organizations like ICANN, Microsoft, and PacketForensics have stepped up to make disks or chassis available (Thank you!).
  • Be cognizant of operational costs. As disk drives increase density over time, the burden of storing data goes down, but one still needs to budget for continuous operation of the storage. It’s more sustainable to allocate a fixed time for storage logs to be available online with a well-defined deletion policy so that researchers who may want or need a longer-term access to the data will know that they need to maintain their own local copy. We can also make arrangements with organizations like DHS PREDICT for sinkhole data or DNS-OARC for DNS data.
  is amazing technology for long term archiving. If you dump data to tape or a disk to take offline, you risk not being able to retrieve it later because you don’t know if it’s gone bad while offline. One needs to regularly verify the data (a process ZFS calls “scrubbing”) to make sure drive or disk sector failures are found in time to ensure the data continues to be available. The checksum operations performed when data is written have helped us discover drive issues before the drives failed or even before SMART technology can predict a failure. We’ve preserved data even through encountering flaky SAS controllers and multiple-disk failures – sometimes two disks in the same RAID set. We now utilize RAID-Z3 where we can.
• Sometimes storage systems can become too large. We once spread a ZFS file system across several disk chassis, and when a chassis or cable started having issues, it caused problems. If a file system is limited to a single chassis, it is possible to move the data or chassis around between servers. As the number of disks within a RAID group increases, so does the risk of a multiple-disk failure causing problems with availability.
• Utilizing a date-driven hierarchy for directories (

YYYY/MM/DD

Conclusion

Farsight Security is proud of our past and current public-private collaborationwith law enforcement, researchers, and Internet security organizations that putDNS and sinkhole data to good use. We look forward to making resourcesavailable for similar future efforts. If you’re a researcher that needs dataor an operator that wants to make sinkhole data available to researchers and/orindustry on a real-time basis, we encourage you to contact us.

Eric Ziegast is a Senior Distributed Systems Engineer for Farsight Security,Inc.