Quadrants Considered Harmful

Perhaps the most consequential task of any enterprise security team involves selecting vendors for cyber defense. Not unlike choosing a spouse, the selection of cyber security vendors can lead to mutually agreeable partnerships – or it can lead to disaster. It is my heartfelt belief that so-called quadrants and waves significantly reduce the likelihood of proper vendor selection, and are thus harmful to our cyber security community

Criticism of quadrants is certainly not new, with most previous concerns focusing on a lack of business transparency. Perhaps surprisingly to the reader, business operations clarity is not my beef here at all. The presence or absence of firewalls between consulting and advisory services does not influence my thinking in the least. Rather, my criticism is more visceral, as it relates to the design and implication of using quadrants to influence source selection in cyber security. I believe quadrants are harmful and here are my arguments:

My first concern involves the ratings accuracy implied by the precise movement one often sees in a company’s rating from one year to the next. It is not uncommon for a vendor to see a physical shift of, say, 600 microns up and 800 microns over from a previous year’s rating (a human hair is about 75 microns). It’s as if some well-calibrated instrument were automating the precise location on a fine graphical display. Unless quadrant writers are willing to explain the basis for such tiny pixilation, I can only conclude that the movement is arbitrary at best and scientifically dishonest at worst.

A second concern is the arbitrary inclusion or exclusion of vendors from a category. Certainly, with limited space on a printed page, the idea of including hundreds of choices would not work. But some categories do include a larger set of options, and by excluding vendors – almost always smaller ones, the quadrant misrepresents the size of the solution set. It would be like producing a quadrant of pediatricians, and only including ones with big waiting rooms and large parking lots. If you’re like me, you might think differently, perhaps wanting your children cared for by a doctor with a small practice, tiny patient list, and cozy waiting room.

A third issue with quadrants seems the most troubling of all – and is perhaps the most controversial of my claims. The problem, as I see it, is that any quadrant or wave offers teams with an excuse to avoid the hard work of source selection. Now, I can already hear the pushback: “We use the quadrant to complement our selection, not to direct it.” And yes, if this is the case, then I congratulate you on your decision, albeit subject to my criticisms above. But many teams rely solely on these thin constructs to select vendors, especially in middle and smaller markets. Quadrants are a crutch for teams that lack confidence to justify their decisions based on insights from diligent research.

Look, I know that vendor selection is hard work, and that many teams do not have the bandwidth to go through all the possibilities. I also understand the visual appeal of a simple grid with vendors scattered across a playing field showing winners and losers for all to see. But let’s face it, the selection of security vendors is akin to how nations invest their precious defense budgets. So, if you are doing enterprise security, then you are doing God’s work protecting our society, and you must up your game.

If you agree with the above arguments, then there is only one reasonable conclusion to draw. There is only one action plan for the entire enterprise security industry to rally around immediately: No more quadrants or waves for source selection. Period.

Let me know what you think.