Threat Hunting: How to Gain the Most Value

Sean Mason, Director of Cisco Incident Response Services and
Jeff Bollinger, Investigations Manager, Cisco Security Incident Response Team (CSIRT)

As security practitioners who continuously look for adversarial malice, one of the questions we are asked frequently is: What’s around the corner? Threat actors evolve over time, so how do we know not only what they’re doing now, but also what’s next? And if things are quiet and we’re not observing any incidents, does that mean that everything is under control? Or are adversaries simply retooling?

To help answer these tough questions, we have threat hunting. The objective of this ongoing exercise is to find and eliminate adversaries that have penetrated defenses and are yet to be detected. Essentially, it’s a shift in mentality. Instead of waiting to respond to an incident after it has triggered an alarm, we’re turning over some rocks to find things we don’t know yet.

As explained in Cisco’s recent report, “Hunting for Hidden Threats,” threat hunting is one more tool in the incident responder’s arsenal. It’s not a silver bullet. But — based on our own 30 years of combined experience mitigating threats, not to mention the whole of Cisco’s experience — we believe it’s an essential component of making security foundational.

How valuable to you is the ability to keep your organization’s data from being stolen or locked, or to keep your organization’s name out of the headlines for a breach? If you can stop even one attack successfully, then all the time and money you’ve invested into threat hunting is worth its weight in gold.

Benefits of threat hunting

Although the ultimate objective is to get ahead of adversaries by finding and expelling them before they cause damage, threat hunting has many other benefits, some of which are:

Improving security operations: While threat hunting itself can sometimes be arduous, you can use it to improve efficiencies in other areas. Once you develop techniques and ways of discovering malicious activity, commoditize and operationalize that by creating playbooks as well as automating some of your day-to-day incident response. At Cisco, for example, our incident response team has more than 400 unique playbooks, many of them informed by our threat hunting activities. We use these plays regularly to look for suspicious activity and to free up analysts’ time.

Understanding your environment: Let’s say you’re a new CISO who needs to get a better picture of what’s going on in your network. A threat hunt, or a compromise assessment, is a good way to understand what you’ve inherited and have signed up to defend. The end result is concrete evidence that you can take to your leadership and ensure you have adequate resources to secure the organization. The hunt can prove that the threats are not just theoretical and are actually lurking inside your ecosystem.

Hardening the environment: From a day-to-day perspective, identifying gaps in security gives you the opportunity to remediate and fix larger problems. As you’re doing hunts, you’ll inevitably discover weaknesses that threat actors can exploit. Apply the knowledge you’ve gained through threat hunting to proactively improve tooling and strengthen the overall security posture.

What it takes to be successful

There are many components to a successful threat hunting program, but the ones that we can’t stress enough include access to the data, a diverse team, and the right mindset.

The importance of high-quality data is obvious, but you may be surprised how big a challenge access can be. We commonly find a lack of necessary data during threat hunts for our customers — and even in our own environment.

Instead of treating a data-access problem like a dead end, think outside the box. Can you look at things differently? Can you use a different set of network logs? And just as important, turn this into an opportunity to improve the outcome next time and go the extra mile to collaborate with those teams that can give you better data.

Which brings us to the people component. There are two aspects to it, and one is the importance of building relationships across teams. Especially those impacted by your security activities, such as the network admins and developers. The other side is the people on the hunting team. Success requires diversity of thought. Include individuals who can think creatively and look at the world a little differently, rather than only thinking in ones and zeroes. We find threat hunters from a variety of backgrounds — even nontechnical.

This also helps you hunt with the right mindset. It’s hard to be objective when you’re living and breathing your security environment day in and day out, especially if you’ve architected it. Taking a step back and asking what you may be missing is not easy. A diverse team that both designs and executes the hunt gives you new perspectives.

Jumping in

Besides the right people, you need the right technology and processes. You may already have a basic foundation you can build on — chances are, you’ve been doing threat hunting without even knowing it. If you’ve ever investigated attacks to try to understand what happened, you’ve been answering some of the same questions and following some of the same steps that hunters do.

A deliberate program, however, does take time to develop. Start with small steps and easy, tactical data sources, then build from there. Don’t make the mistake of throwing a bunch of data sources in at once, or you’ll run into challenges. You don’t even need complicated tools to get off the ground, because you can discover malicious behavior with OS event logs or logs your sysadmin keeps for troubleshooting purposes.

One final thought. There’s a misconception that only larger organizations can implement a threat hunting program. In reality, threat actors don’t concern themselves with size and are looking for easy targets — smaller organizations can benefit just as much, if not more, from getting ahead of these threats. If you don’t have in-house resources, outsource to an expert consultant. And if you already have an outside IR team on retainer, start the conversation about what it would take to proactively look for adversaries.

Want to learn more about establishing a threat hunting program? Download the recent Cisco Cybersecurity Series report, “Hunting for Hidden Threats: Incorporating Threat Hunting Into Your Security Program.”