Lessons Learned from SUNBURST for Threat Hunters

Introduction

The incident that was initially called “the SolarWinds hack,” but whose scope has steadily increased as we’ve learned more about it, is significant enough that it’s safe to say books will be written about it. Some of them may wind up as bestsellers. More properly described as a constellation of events rather than one, it offers a lot of lessons for everyone involved in information security; whatever your particular area of interest, there’s likely to be some part of the constellation that is pertinent. At DomainTools, not surprisingly, the infrastructure used by the adversary for command and control is of particular interest. Joe Slowik from our research team has written several excellent blogs on it, and the timeline below comes from one of those.

On a panel discussion I participated in shortly after the news broke in December, the question arose as to whether threat hunting could have uncovered this incursion in its earliest stages. The consensus was that it couldn’t have, but I was not convinced. I decided to convene a panel of our own to explore this and related questions about what reasonable expectations should be around the potential for hunting to discover supply chain breaches.

My co-panelists for the DomainTools discussion were Joe, his research team colleague Chad Anderson, and Lead Sales Engineer Taylor Wilkes-Pierce. Each of these three brings a valuable perspective on various threat hunting methods. We explored four main areas:

• What type of hunting teams should be doing now if they’re unsure of whether they’ve been compromised by the SolarWinds event
• The role of adversary infrastructure-based hunting
• Recommendations on what are likely to be the best ROI hunting/incident response activities, especially given the flood of indicators and TTPs in the wake of the event
• Whether threat hunting could have caught this incursion in its earliest stages

There was a fifth question as well, which came up during our audio check before the webinar: what can teams do to make sure their build systems are clean? More on that at the end of this post.

What Hunt Teams Should Be Doing Now

On the first point—what type of hunting teams should be doing now if they’re unsure of whether they’ve been compromised by the SolarWinds-related incursion—there were a few major points that the panel identified:

• While it is likely a bit late in the game for this, it’s worth checking for any traffic to the (now neutralized) C2 domain avsvmcloud[.]com
• A second stage of the incursion involved Cobalt Strike beaconing. This lends another set of artifacts for hunters to search for. (This Microsoft Blog is an excellent resource for the Cobalt Strike component of the incursion)
• In general, enrichment of Internet observables at scale (such as all or some large proportion of your new DNS resolutions) is key toward starting to unravel potentially suspicious activity.

The Role of Adversary Infrastructure-Based Hunting

The question about adversary infrastructure hunting, which is treated fairly extensively in the blogs by Joe referenced earlier, raised a couple of valuable points for hunters to consider:

• DNS traffic to avsvmcloud[.]com is not just an indication of SUNBURST activity, it also provides a useful example of what DNS abuse for C2 looks like. Hunters should consider building detections for abnormal DNS traffic. (This SANS paper, while a few years old, has great information about detecting DNS C2.)
• Other than a relatively low number of domains for SUNBURST and for the Cobalt Strike beacons, there is not a huge amount of infrastructure to explore for this particular incursion (as it is presently understood). The panel agreed generally that some incursions or attacks will involve more network observables than this one did.
• Hunting for new assets being created by an existing actor can be trickier than it used to be—adversary OpSec is better now, plus more registration information is obscured. That said, patterns such as combinations of hosting providers, SSL info, IP and Name Server data, etc., still make it possible to keep pace with emerging campaigns.
• It’s also about more than domains and IPs; Chad observed that it’s increasingly important to look at the service layers (such as TLS/SSL, onscreen content, etc). As he said, “Something always has to call out to a service,” and there are always some clues available from that traffic.

Tips on the Best Hunting/Incident Response Activities

One of the objectives of the panel discussion was embodied in our third question, about which techniques would give hunt teams (or any SOC personnel doing ad-hoc hunting) the best bang for the buck. There were a few suggestions here, some of which fall more into the category of setting oneself up for success in hunting, vs the hunting itself.

• Asset management, and identification of “crown jewels” resources, is critical. Especially in a very large environment, there can be an almost overwhelming amount of data to sift through. Blue teams should give priority to traffic outbound from critical systems; but obviously said systems have to be identified as such, and filtered for in the hunt. Taylor pointed out that monitoring new domain and hostname resolutions from these devices is still one of the best ways to detect dangerous activity.
• Network segmentation plays a major role in this as well. Chad pointed out that flat environments, especially large ones, have a much bigger battle ahead of them than those that are well segmented.
• Network traffic analysis is a well understood technique, but it isn’t done by as many shops as one might expect. Combined with the above two preconditions, the focus for traffic analysis becomes more manageable. Further filtering can make it even more so; for example, filtering out DNS queries to the Alexa top million domains will substantially reduce the amount of DNS traffic needing to be analyzed. Further, there are techniques using regular expressions that can catch unusually-formatted DNS requests. Such rules could have caught the victim-identification encoded A record lookups from SUNBURST.

Would Threat Hunting Have Caught This Incursion?

The fourth question posed was: could SUNBURST have been detected by conventional threat hunting methods? The consensus of our panel was yes, but a qualified yes. For one thing, we wanted to acknowledge that the adversary practiced extremely good opsec in this incursion. Hunt teams that didn’t discover this incursion independently (i.e. before indicator lists were published) have no cause for shame. At the same time, much of what we addressed during the discussion, such as network traffic analysis and monitoring of crown jewels assets, could reasonably have been expected to catch some of the signals of the incursion, even if these signals would not have told the team much about the nature of the event (absent external context given after the news broke widely).

Joe also touched on the importance of understanding what an adversary’s route into your environment would have to look like; this is where threat modeling can really inform a lot of valuable hunting activities. He further pointed out that identifying behaviors such as remote logon (particularly odd logon patterns) and other lateral movement signals is both important and broadly applicable, in that it can uncover a range of activities from state sponsored, genuine APT actions to commodity malware or ransomware.

Tips for Keeping the Pipeline Clean

The question that came up during our audio check before the webinar, and which was also asked in close paraphrase by an audience member was this: how does one go about ensuring that compromised or flat-out malicious binaries are not part of the pipeline? Chad offered that something like a Git commit pre-hook where everything is hashed and then submitted to another server that verifies that it was built as intended, could be one methodology. However, he also acknowledged that speed of development is at odds with measures such as this, so it’s important to make any such checking as seamless as possible.

There’s no doubt that SUNBURST and its related activities will continue to be in the news for a while, and that more organizations will yet discover that they have been affected. We salute the teams working hard to mitigate this and related incursions.