I learned physics in the middle of nowhere: Dickinson College, located in Carlisle, Pennsylvania. The small school was my home from 1979 to 1983, and apart from being a lawyer and doctor factory, it has also carried the burden of being the Alma Mater to our nation’s worst president ever – James Buchanan. As you’d expect, many Dickinson drinking glasses have been raised recently to toast passing the baton to Wharton.
During those years, my growing interest in computing was tolerated and even supported by Priscilla Laws – one half of Dickinson’s so-called Laws of Physics: Professors Priscilla and Ken. She looked the other way, for example, when my projects were 99% software and 1% physics. She was also our tour guide to local industry, including a visit to Three Mile Island in nearby Harrisburg just after the big accident. (I still have a Friends-of-TMI button.)
Little did any of us know that just a few years later, one of the most important computer security organizations of all time would be founded on Walnut Bottom Road, just a stone’s throw from my SAE Fraternity house. The National Computer Security Association (NCSA), founded by the legendary Dr. Peter Tippett, emerged in 1988 and arguably became one of the most consequential security organizations during our industry’s formative years.
So, three decades later, it was pure delight for me to re-connect with the team that continues to carry the torch for what is now called ICSA Labs. Still located in the middle of Cumberland County, several of their longtime experts – Dave Archer, Greg Wasson, Darren Hartman, and Jack Walsh – crowded into a conference room and brought me up-to-date. Let me share what I learned from the guys:
“Our team mission has been and remains security testing,” explained Jack Walsh, who program manages the security testing work at ICSA Labs. “We offer companies an independent, third-party source for demonstrating assurance in their cyber security products. The goal is to establish certification of a given product with respect to the formal security criteria established by our ICSA team.”
This concept sounds simple, but is anything but. Demonstrating compliance of a product against a set of formal criteria has been elusive, ever since creation in the 1980’s of the original Trusted Computer System Evaluation Criteria – aka the Orange Book. Nevertheless, the ICSA Labs team has evolved a practical test methodology that seems to work well for vendors, enterprise teams, and any other entities requiring insights into security products.
The ICSA Labs security test process starts with an initial installation of the product of interest into the ICSA test environment. This involves delivery from the vendor of their solution, usually some combination of hardware and software. The ICSA Labs team then subjects the product to a battery of testing, using the requirements in their criteria as a guide. These include the functional capabilities deemed critical for proper operation of the product.
The test process is designed to be continuous, to ensure compliance throughout the product lifecycle, and to avoid the moment-in-time problem of one-time testing. Periodic test updates range from monthly, to quarterly, to annually, depending on the specific product and how frequently its functionality is evolving. On-going testing is obviously more work, but it provides improved assurance for enterprise users.
I asked the team whether they include advanced exploit testing, including penetration testing. “Our criteria are designed to include as many applicable test cases as possible, including ones designed to address common exploits,” explained Dave Archer, engineering manager at ICSA Labs. “While we don’t do penetration testing, we do exercise the functionality of the products pretty thoroughly.”
Testing services from ICSA Labs cover advanced threat defense, anti-malware, Internet of Things (IoT), mobile devices, network peripherals, firewalls, network IPS, SSL-TLS, and web application firewalls. The criteria requirements are public, easy to understand, well-vetted, and supporting of pass-fail results. Special emphasis areas are also supported, including for Electronic Health Record (EHR) technology.
Obviously, new technical challenges have begun to emerge for product security testing – and I discussed this with the team. For instance, as software is delivered continuously via DevOps processes, where new features are being generated more quickly than ever, the idea of delivering an appliance to a lab for off-line testing seems quaint. The team agreed that evolving their test process to support Agile lifecycles is a priority.
In addition, the team delivers pass/fail results to solution vendors, which makes it tougher to nuance a given functional feature. Furthermore, any vendor who goes through the ICSA Labs process who doesn’t like the end-result has no obligation to report their score. This approach is certainly friendly – and remember that this team works in God’s country – but it might be more fun to see a few C-pluses and D-minuses reported. But I get it.
ICSA Labs competes with NSS Labs, and I have no idea which test process is better (perhaps because I haven’t really dug into the competing offer yet). But I can tell you that ICSA Labs is comprised of a fine, honest, hard-working team with decades of experience. If you see the ICSA Labs certification on a product, you can be sure that someone from Cumberland County worked hard to make sure the darn thing works properly.
Give them a call and let us all know what you learn.