Big Data Needs to Equal Big Results

‘Training for Success’with Bill Russell

Bill Russell has worked with clients globally from a variety of industries including defense, telecom, banking, manufacturing, automotive, aviation, medical equipment, healthcare, and universities. Through the use of workshops, presentations, and interventions, he strives for quick, collaborative implementations that prioritize actions on the issues that yield the fastest and largest payback and encourage multiple cycles of learning. In other words, Bill believes in training for success. In this column, Bill will offer words of advice on how to get the most out of your training sessions. A firm believer in the power of reading and continuing education, Bill will also offer a recommended read at the conclusion of every piece.

Situation: Big data has become too complex for its own good.

Big data is a hot topic in executive circles these days. As it is usually defined, big data is a collection of data so large or complex it becomes difficult to process using traditional database management tools or data processing applications.

Bill’s Approach: Learn to maximize your use of big data to solve business problems.

Big data is in action all around us. Marketers use it to spot the next big thing. National Security depends on it to identify dangerous activity in a sea of transmissions or actions. It can be used to spot correlations in business trends, prevent diseases and track road conditions.

Most analytics people will tell you more data is always better. Heap it up. In my experience, most individuals and organizations err on the side of doing too little with the data they have, wasting the opportunity to learn from collective experiences. So any new trend that raises awareness about big data’s potential and gives people new tools to work with is good, but
I must offer a few disclaimers.

There are a few of lessons learned by those using data extensively that could benefit us all:

  • First, define the issue you wish to learn about. Understand what is it, and what isn’t it.
  • Predictions and models are only a means to an end. George Box had it right: all predictions are wrong, but some are useful. Assumptions are key. Map them carefully. (Remember the recent economic meltdown?)
  • People are great at pattern recognition. We will look for patterns in all sorts of areas. The problem with noisy data – like the kind in big data sets – is that people will begin to mistake the (abundant) noise for the (faint) signal. Hence the false alarms and conflicting news broadcasts.
  • The sheer quantity of data might numb some people into complacency. “With so much data, who needs theory?” Interestingly, in fields where there is little underlying theory (economics, politics) there is much less progress in making better forecasts than in fields where there is some theory (weather, sports).
  • Nothing replaces careful thinking about the situation at hand. Map the necessary assumptions. Determine the right metrics. Determine precision and accuracy in making those measurements. Then collect all the data you can.

Recommended Reading: The Signal and the Noise by Nate Silver

Nate Silver built an innovative system for predicting baseball performance, predicted the 2008 election within a hair’s breadth, and became a national sensation as a blogger—all by the time he was thirty. The New York Times now publishes, where Silver is one of the nation’s most influential political forecasters.

Drawing on his own groundbreaking work, Silver examines the world of prediction, investigating how we can distinguish a true signal from a universe of noisy data. Most predictions fail, often at great cost to society, because most of us have a poor understanding of probability and uncertainty. Both experts and laypeople mistake more confident predictions for more accurate ones. But overconfidence is often the reason for failure. If our appreciation of uncertainty improves, our predictions can get better too.