“Whitelisting” at the OS Level

Over the last several years, the endpoint detection and response (EDR) marketplace has seen enormous growth. With dozens of dedicated EDR platforms to choose from, and fundamental endpoint protection platforms vying for a spot in security teams’ toolboxes, it can be hard to sort through the noise and determine which features and functionalities are the best fit for your company. EDR has become such an important space because of its enhanced capabilities for threat hunting and investigation, but antivirus and anti-malware technologies remain fundamental pieces of the puzzle.

In the past, cries of “Antivirus is dead” have been heard ringing through the halls of security conferences. But the reality is that AV/anti-malware do catch a lot of known threats at the gate. In other words, for what it can deliver, companies would be foolish to eschew traditional endpoint protection entirely when it stops a high percentage of attacks.

That said, it’s often not the low-hanging fruit that keeps security teams up at night. If we look at all the headline breaches over the past few decades, every victim company had firewalls and AV deployed. As a result, the security industry has turned to protecting against unknown malware, “advanced” malware, and APTs. And while tools and techniques have improved with each new market space, due to the law of large numbers and over-reliance on blacklisting technologies, malware attacks continue to be effective. The result was the emergence of the EDR market in the mid-2010s. In the late 2010s, a new type of EDR company was launched.

A 180° spin

Nyotron (pronounced “new-tron”), an endpoint protection company out of Santa Clara, CA decided to tackle the malware problem from what they say is “the exact opposite point of view from traditional EDR.” The company’s endpoint prevention and response (EPR) technology was built on the idea that identifying malware attacks and methods is an infinite problem, i.e., the attackers will always have the upper hand because there are limitless types of malware that can be written and immeasurable ways a threat actor can attack an endpoint. “But that is a battle you can’t win,” said Rene Kolga, VP of Product Strategy at Nyotron during a recent briefing. “It’s just not possible to enumerate infinity.”

Instead of focusing on how bad guys break in and the behaviors they exhibit in doing so—what Kolga described as the “negative security model”—Nyotron’s flagship product, Paranoid, uses a “positive security model” to define “good” and rejects everything else. This sounds, then, an awful lot like application whitelisting (a.k.a. application control) from more than a decade ago. Security pros know, though, that the problem with traditional whitelisting has always been how difficult and manually intensive whitelisting is to manage. Moreover, application whitelisting does not address fileless malware or living-off-the-land techniques. At its best, however, application whitelisting is not only valid, but the results are highly positive despite its downfalls.

Operating system behaviors

Rather than focus on applications as the means by which companies control what is allowed to run on the network, Paranoid contains the whitelist of operating system (OS) behaviors—anything related to file systems, process management, registry modification, networking, and partition modification. “What this does,” said Kolga, “is apply the best parts of whitelisting and remove the management overhead.” It also reduces the attack surface; OS-level behaviors are finite and fairly static, while the number of applications executing on a network can run in the thousands and is quite dynamic.

By looking at OS-level behaviors, the system bounds what it needs to look for, and system administrators don’t have the headache of constant changes. Paranoid contains the map of the legitimate behavior of the OS—normative routes of the operating system in the form of sequences of system calls—without disk scanning, static file analysis, cloud reputation lookups, sandboxing, or any of the other higher-level processes that make some detection and monitoring tools resource-heavy and noisy with alerts and false positives. “Kolga explained, “Paranoid doesn’t care what the malware is; it only looks at the system call sequences that are in-line with the designed behavior of the OS. Because this legitimate behavior on the OS doesn’t change frequently, customers get precise visibility and protection without the management overhead.”

Reducing the problem space

Despite the new category Nyotron created (endpoint prevention and response), Paranoid can run in block mode or monitor-only mode. This is bound to be a welcome feature, as many organizations might not be comfortable with auto-blocking at the OS level before having some time in monitor mode. Kolga said, however, that most customers do run in block mode, because, “if a user takes advantage of block, it’s happening at a granular level so it doesn’t disrupt the entire system and has minimal chances for false positives. It’s a small problem space compared to attempting to enumerate all “badness” in the world.”

Another consideration is where Paranoid might fit into security or operating teams’ budgets. Although one could surmise that EDR/EPR is a replacement for AV or firewalls, Kolga says every one of their customers has both AV and firewalls deployed. The fact is, AV and EPR (i.e., Paranoid) tackle security from different ends of the spectrum and are thus complementary, not redundant. While Nyotron appears to be the only company approaching malware prevention from this unique perspective, at some point in the future it would not be surprising to see OS behavioral whitelisting embedded in the more common AV or EDR solutions. Just like EDR turned into EPP and now most EPP suites include EDR capabilities, the future of malware management at the endpoint could also include EPR