There are fewer than 300 white lions living on the planet.* To say they are rare and exceptional is an understatement.
Today, people with the multi-disciplinary skills necessary for advanced information security are almost as rare as white lions. In just the areas of threat discovery and breach detection, IT security teams need folks who are knowledgeable in network & host security, threat intel, forensic analysis, and big-data science.
Finding this diverse knowledge set within a group of IT security professionals is rare. Places like Aetna, Bank of America, Facebook, Google, and Netflix have many of these exceptional skills in-house. However, ‘productizing’ security knowledge and operating skills is extremely difficult.
During the past 6-8 years, considered the first phase of big-data security, a few specialized consulting firms found success providing security skills as professional services. As an example, Palantir built a fast-growing consulting business – now privately valued at $14B USD – by essentially offering “white lions” for hire. These highly sought after consultants command eye-popping rates of $1,000-$3,000 per hour.
What do clients get for those consulting rates? From what I’ve seen, they get help exploring some important questions as described below.
Important big-data security questions:
- Can analytics and elastic search software deliver faster and human-interpretable security insights?
- Can big-data clusters like Hadoop process massive amounts of machine-generated data to expose anomalous and potentially malicious threat patterns?
- Can advanced network forensics, combined with log data, be processed with machine learning algorithms to provide new insights that security information and event managers (SIEMs) do not address?
The good news is that the answer to all these questions is an emphatic YES.
The industry benefits from having both consultants and early users of big-data experiment and blaze new trails using rapid prototyping and custom code delivery. Already, a select group of deep-pocketed companies have proven to themselves that new approaches using big-data analytics can help tackle some of the toughest problems in threat detection. However, those same companies are now fatigued by the outrageous cost of adding new security features and capabilities. Plus, they carry the burden of maintaining bespoke software stacks often stitched together by outsiders who have moved on to other projects. And good luck to the companies without deep pockets, but who still need advanced IT security.
Customers have figured out that big-data works for security but the lack of high-quality products in this area is a barrier for broader adoption. As a result, IT security teams of all sizes have begun the DIY (do-it-yourself) approach because so few choices are available in the market.
The first few steps commonly taken with roll-your-own big-data security:
- Leverage log data from their legacy SIEM or Splunk environment (machine generated data sources)
- Leverage network packets from various port taps or network sensors (live traffic data sources)
- Deploy a Hadoop cluster using Cloudera or HortonWorks (scale-out data processing engine)
- Add some 3rd party sources for cross-checking known malware or bad actors (additional verification)
- Add ElasticSearch for faster indexing and an analytics package for visualization (human interpretable UX)
This is a good starting point, but still only the tip of the iceberg.
Further, do IT teams really want to maintain this complexity themselves? To me, this is like an IT department rolling their own firewall, IPS, router, or database. Yes, it can be done but is this the best use of IT and security resources?
As a security industry, we have graduated from the phase of drive-by software consultants selling to first-time customers of big data. We are beginning the next phase of threat discovery: where cohesive products built by innovative engineering teams step forward. When these new security products emerge, organizations will experience faster feature innovation, improve their ability to discover high-priority threats, and reduce the need to seek out those rare “white lions” for hire. Rolling your own big-data security stack might be the best approach for some set of customers. My guess is that the bulk of the industry will want to use their in-house talent for higher value initiatives. Those companies will wisely leverage off-the-shelf software that works as advertised, and the best product teams are likely to come out on top.
With that in mind, I am keen to watch Niara and several other new startups in this emerging category of big-data threat discovery. My firm Venrock has been fortunate enough to lead Niara’s recent $20M funding alongside existing investors NEA, Index Ventures, and Aruba’s CEO Dominic Orr. The security world will soon learn more about the innovative products coming from this special team of “white lions” at Niara.
Exciting times ahead for the security industry.
*Global White Lion Protection Trust – see question #17