big data security white lions

White lions of big-data security

rare white lions

There are fewer than 300 white lions living on the planet.* To say they are rare and exceptional is an understatement.

Today, people with the multi-disciplinary skills necessary for advanced information security are almost as rare as white lions. In just the areas of threat discovery and breach detection, IT security teams need folks who are knowledgeable in network & host security, threat intel, forensic analysis, and big-data science.

Finding this diverse knowledge set within a group of IT security professionals is rare. Places like Aetna, Bank of America, Facebook, Google, and Netflix have many of these exceptional skills in-house. However, ‘productizing’ security knowledge and operating skills is extremely difficult.

venn diagram big data security

During the past 6-8 years, considered the first phase of big-data security, a few specialized consulting firms found success providing security skills as professional services. As an example, Palantir built a fast-growing consulting business – now privately valued at $14B USD – by essentially offering “white lions” for hire. These highly sought after consultants command eye-popping rates of $1,000-$3,000 per hour.

What do clients get for those consulting rates? From what I’ve seen, they get help exploring some important questions as described below.

Important big-data security questions:

  • Can analytics and elastic search software deliver faster and human-interpretable security insights?
  • Can big-data clusters like Hadoop process massive amounts of machine-generated data to expose anomalous and potentially malicious threat patterns?
  • Can advanced network forensics, combined with log data, be processed with machine learning algorithms to provide new insights that security information and event managers (SIEMs) do not address?

The good news is that the answer to all these questions is an emphatic YES.

The industry benefits from having both consultants and early users of big-data experiment and blaze new trails using rapid prototyping and custom code delivery. Already, a select group of deep-pocketed companies have proven to themselves that new approaches using big-data analytics can help tackle some of the toughest problems in threat detection. However, those same companies are now fatigued by the outrageous cost of adding new security features and capabilities. Plus, they carry the burden of maintaining bespoke software stacks often stitched together by outsiders who have moved on to other projects. And good luck to the companies without deep pockets, but who still need advanced IT security.

Customers have figured out that big-data works for security but the lack of high-quality products in this area is a barrier for broader adoption. As a result, IT security teams of all sizes have begun the DIY (do-it-yourself) approach because so few choices are available in the market.

The first few steps commonly taken with roll-your-own big-data security:

  1. Leverage log data from their legacy SIEM or Splunk environment (machine generated data sources)
  2. Leverage network packets from various port taps or network sensors (live traffic data sources)
  3. Deploy a Hadoop cluster using Cloudera or HortonWorks (scale-out data processing engine)
  4. Add some 3rd party sources for cross-checking known malware or bad actors (additional verification)
  5. Add ElasticSearch for faster indexing and an analytics package for visualization (human interpretable UX)

This is a good starting point, but still only the tip of the iceberg.

Further, do IT teams really want to maintain this complexity themselves? To me, this is like an IT department rolling their own firewall, IPS, router, or database. Yes, it can be done but is this the best use of IT and security resources?

As a security industry, we have graduated from the phase of drive-by software consultants selling to first-time customers of big data. We are beginning the next phase of threat discovery: where cohesive products built by innovative engineering teams step forward. When these new security products emerge, organizations will experience faster feature innovation, improve their ability to discover high-priority threats, and reduce the need to seek out those rare “white lions” for hire. Rolling your own big-data security stack might be the best approach for some set of customers. My guess is that the bulk of the industry will want to use their in-house talent for higher value initiatives. Those companies will wisely leverage off-the-shelf software that works as advertised, and the best product teams are likely to come out on top.

With that in mind, I am keen to watch Niara and several other new startups in this emerging category of big-data threat discovery. My firm Venrock has been fortunate enough to lead Niara’s recent $20M funding alongside existing investors NEA, Index Ventures, and Aruba’s CEO Dominic Orr. The security world will soon learn more about the innovative products coming from this special team of “white lions” at Niara.

Exciting times ahead for the security industry.

*Global White Lion Protection Trust – see question #17

dougdooley456White lions of big-data security


  1. Pingback: White lions of big-data security : Venrock

  2. Anonymous

    Doug — great post. First of all, I really like the white lion comparison; the Venn diagram you have of the needed skillsets really aligns well with what the average enterprise experiences. I think the second half of the post is very strong, where you’re teasing out the key components of the roll-your-own approach.

    The only thing I might add is in the paragraph where you mention Palantir offering white lions for hire. I would imagine that they have gotten much better at this over the past few years, but based on their website Palantir seemed to prefer recruiting highly educated recent college graduates with strong computer science backgrounds but no domain experience. While you might be lucky enough to work with very strong data scientists and one or two white lions in the beginning, I would imagine these resources would be quickly assigned to new accounts (read: new breaches) and the apprentice resources replace the domain experts. I think your post calls out what the industry needs — technologies that cover the domain expertise gap.

    However, what you don’t really mention is that Palantir wows people with their analyst interface. Palantir does have a robust technology stack that demonstrates well. I think it’s really the combination of their productized big data technology PLUS these talented computer science resources that made them successful… Palantir sees themselves as a technology company and not a consulting company.

  3. Param

    Agreed that white lions are rare and it is difficult to find talent that is top-notch in all four areas mentioned above. However another approach to consider here might be to identify a different species, someone that has domain expertise in one of four areas and is both a quick learner and innovative.

    Due to rapid development and pace of change in all four areas, it would be difficult for any professional to stay abreast with all the advances and therefore will slowly lose touch. Therefore, in my opinion, the goal for an upcoming organization like Niara should be to identify and build a team of quick learners and innovators from those four domains, allow them to brainstorm for best ideas and execute on those together, as a team – which is even better than a single ‘white lion’.

    As Steve Jobs said, “My model for business is The Beatles. They were four guys who kept each other’s kind of negative tendencies in check. They balanced each other and the total was greater than the sum of the parts. That’s how I see business: great things in business are never done by one person, they’re done by a team of people.”

Leave a Comment