Enterprise Cyber Threat Hunting Using Big Data

The proper role for humans in real-time cyber security has not always been crystal clear. My belief two decades back was that automation should feed prioritized alarms to friendly operators wearing headsets (think colorful ADT brochure), and that customers would receive timely notifications whenever something was amiss in their network. It was a logical, attractive view of cyber operations – and it was wrong. The problem is that the complexity of modern computing prevents simplistic conclusions from being drawn about cyber threats based on IDS alarms. Instead, contextual analysis is required to determine accurate posture and recommended remediation. Most of this work is done by humans, even though considerable focus is on-going in the cyber community to automate this task using machine learning and AI. Amidst this evolution, an interesting new role has emerged in our community, one now referred to as hunting – and I believe this to be one of the more sensible advances in our industry. The idea is that the human hunter, an expert trained in the use of Big Data analytic tools, has the responsibility to fill in the gaps around automation. This task includes developing insights that can be used to improve the SOC tools. I first became aware of this exciting new trend years ago through discussions with my good friends at Sqrrl. I recently had the opportunity to spend time with their fine CTO, Adam Fuchs, to learn more about cyber hunting and threat analytics in the enterprise.

EA: Adam, is security analytics anything more than just correlating collected data?

AF: Absolutely. Security analytics is the application of advanced algorithms, including supervised and unsupervised machine learning and graph algorithms to identify threats that evaded detection or proper prioritization by other security systems. Many of these algorithmic techniques have been around for a while, but one of the major advances for security analytics today is the ability to deploy at scale across massive amounts of data. However, it is not enough to just correlate and automate analysis at scale. Textbook application of machine learning techniques frequently produces groundbreaking insights, like “http traffic is often seen on port 80,” thus leaving a fair amount to be desired. To truly impact the security domain, analytics require structure and context. This includes structure in the form of behavior and attack models, and context in the form of broader prospective from multiple sensors, risk analysis, and feedback. With appropriate structure and context, security analytics is an incredibly valuable tool for hunting, detection, and forensics.

EA: How hard is it for security analysts to learn to hunt attacks? Do analysts need to be experts in networking, mathematics, and investigative forensics?

AF: Our assumption at Sqrrl is that our customers don’t need any data science skill sets. Historically, to proactively hunt for threats you needed to have data science skill sets to build custom algorithms to look for anomalies that other tools missed. Our algorithms work out-of-the-box on standard datasets seen in most SOCs. The structure for our analytics comes from extensive modeling contributed by security and networking experts. Sqrrl deployments learn much of the necessary context through observing data feeds from a variety of sensors. From the start, our analytics find interesting behaviors that give insights into what’s happening on the network. Analysts provide feedback on false and true positives over time to hone in on exactly the behaviors that matter to them.

EA: How does the enterprise transition to mobile devices and cloud systems affect the security analytics process?

AF: One of the big changes with increased use of mobile and cloud is that enterprises are starting to give up on the idea of a secure perimeter. With attack vectors in email, web browsing, and countless other common activities, secure perimeters have been a dubious concept for over a decade. Mobile and cloud systems are acting as a forcing function for companies to break old habits and begin adopting more effective tools and techniques. For those of us already in the modern world, the biggest change we see is in our sensors. Mobile and cloud systems make some behaviors harder to spot and other behaviors easier. With well-structured security analytics, we can take advantage of new datasets that can provide additional detail and context into potential attack pathways and attacker TTPs (tactics, techniques, and procedures). As an example, security analytics tools can take logs from cloud access security brokers (CASBs), and correlate behaviors associated with them to look for data exfiltration patterns, and connect those patterns to other TTPs correlated with the same hosts and users.

EA: What sort of trends do you see in cybersecurity vulnerabilities in the enterprise?

AF: In general, we are seeing increased cyber security awareness and better cyber hygiene in large enterprises. However, many attacks do not require exploitation of traditional software vulnerabilities. These exploit-less attacks often take advantage of human vulnerabilities and then move laterally and escalate privileges without the use of malware. This explains why enterprises cannot rely on anti-malware or anti-virus solutions as a sole layer of defense.

EA: Do you think that smaller companies can ever take advantage of security analytics tools directly? Or do they need to rely on managed security service providers with trained staff?

AF: We believe it is critical to still have a human in the loop when conducting threat hunting. Fully automated solutions can only get you so far. As a result, we do see benefits in smaller companies taking advantage of the MSSPs. Recruiting, training, and retaining advanced security personnel are difficult for larger companies, let alone smaller companies. MSSPs can help mitigate this, and more and more MSSPs are now offering specialized threat hunting services to their customers. Beyond just the expertise consideration, MSSPs are also uniquely positioned to correlate attack indicators across multiple companies. This really helps to identify signals in the noise and pick out potential attacks earlier.

EA: Do you think it is realistic for an enterprise to ever hope to detect attacks from advanced nation state actors? It seems like an unfair fight.

AF: No organization can guarantee that a well-resourced, determined adversary will not be able to breach their perimeter security. However, threat hunting and security analytics can greatly assist enterprises in reducing the probability that such an attack will be successful. Sqrrl has assisted a variety of Fortune 2000 companies, government agencies, and MSSPs in detecting these types of advanced nation state actors.