Ben Wuest is a senior member of the IBM Security engineering team and works with clients all over the world. Wuest started as a software engineer for Q1 Labs in 2008 and now holds a top spot as the chief architect and CTO of IBM Security Intelligence.

In this interview, he offers his perspective on how big data will continue to shape the security intelligence and analytics landscape.

Question: How could an organization use big data to help with cybersecurity?

Wuest: Well first of all, in many ways a security intelligence platform is a big data analytics solution. It collects logs from many different sources, adds network flows, performs correlation and applies intelligent rules and analytics to transform millions and even billions of events into practical, near real-time information that customers can use to address advanced threats, fraud and many other use cases. We suggest that clients start there.

But another option is to go even further and set up a big data infrastructure, like Hadoop, for actionable threat intelligence. I recently worked with a client who was building data sets that included their own information and external threat feeds. Their goal was to integrate this data into their security operations to help detect and respond to security problems. They had IBM Security QRadar installed and set it up to communicate with the big data system.

What’s driving these types of projects?

It’s really about the need to continue to increase visibility and protection through advanced analytics. Clients have to solve problems like insider threats and data leakage and are getting creative by combining various data sets to look for patterns.

Want to learn more? Read the IT Executive Guide to Security Intelligence

What advice do you give organizations who are exploring big data?

Understanding the problem at hand is key. Before you start funneling/duplicating data all over the place, take the time to understand exactly what you’re trying to get out of the experiment and step into it lightly. Just dumping data into a large Hadoop infrastructure is going to cause one bump in the road after another, especially with respect to access controls.

Another area to consider is how to organize your staff and resources. You really need to connect your big data scientists and the security analysts. It’s not helpful to send security data over the wire to the big data pool and never see it again, and vice versa.

And third is to understand your platform. I spoke with a CISO who had no idea that his QRadar platform could solve what they needed to do in the short term. Organizations think they always need new tools to do big data analytics. And, you know, often organizations start shopping for new solutions without actually looking at what tools they have.

Is there a standard set of tools that organizations use for this type of analytics?

A big data platform is not a purpose-built security solution, so work has to be done to customize it for what you need it to do. The right security information and event management (SIEM) solution can save some time because it can normalize a variety of data feeds into highly structured data. With an SIEM, a login event is a login event no matter what type of device it comes from.

Without an SIEM, you would have probably 10 times the problems with your big data solution because you would be responsible for deciphering this structured type of data in your platform, which would compound the resource problem.

How can an organization determine where to begin?

Well, the first step is to understand the problem you are trying to solve and what data you need to bring in. Large-scale big data analytics are not meant for real time. They’re more about longer, batch-processing algorithms that implement machine learning techniques. There is a higher level of complexity. There is absolutely a place for big data, but you really need to start by looking at your entire infrastructure and understanding your process, infrastructure, assets and identities. And with the right people, you can do some really advanced analytics — even cognitive security analytics that can potentially improve your team’s efficiency.

That being said, the reality is that probably 40 to 45 percent of security incidents can be solved pre-exploit. You can assess your network, scan it for vulnerabilities, prioritize them, assess your network topology to understand how things are configured and then be able to see when something’s not right. This level of pre-exploit analytics is not something that you’re going to get baked into a big data analytics platform.

Where do you see the industry going in this space?

We first built QRadar, if you look way back, because of all the disparate information on the network. We introduced the analytics to help make sense of it all. Today, with big data, things are starting to branch out again. We’ll continue to evolve the platform with analytics to help our clients stay ahead of the bad guys.

Download the IT Executive Guide to Security Intelligence to learn more