Big Data Alone Doesn’t Always Give the Right Conclusion

Big data is good, but rich data is even better.

That’s the message data guru Nate Silver shared during his luncheon keynote address Friday.

“Size means something but you also want variety … observations collected under a wide variety of conditions,” said Silver, “data that empowers better decisions.”

“If you do read the hype about data and statistics it might seem like it can solve all of our problems,” he said. “Data alone doesn’t always point you to the right conclusion.”

In fact, when he was working during the 2008 presidential election, in which he correctly predicted the outcome in 49 states, Silver used a lot of data, but he combined it with analysis.

Using his 538 method, Silver looked at an average of polls and avoided chasing the “shiny object” of a poll that might have showed something totally different.

He then looked at the combinations that would get the candidate to 270 electoral votes to predict the winner of the presidential race. He also looked at probabilities and not absolutes.

“You can be more confident about a prediction the closer to the event you get,” Silver said.

The explosion of data has led some to say that the scientific method will become obsolete, Silver said. But someone must still put the data into perspective.

For instance, the day of the 2012 election, the conservative-leaning website, The Drudge Report, had more than five times the number of articles showing Mitt Romney leading in the polls than President Obama, when in fact the opposite was true.

“If you had only read this news site you would have been very surprised at the results,” Silver said.

He offered four suggestions for policymakers:

Think probablistically and explore the margin of error of those probabilities. He cited an example of flooding along the Red River in 1997, where meteorologists looked at statistical models and predicted the river would crest at 49 feet—the levee was built at 51 feet. The problem, however, was that the predictive model had a margin of error of plus or minus nine feet. The river crested at 53 feet.

Know where you’re coming from. Silver said the Sept. 11, 2001, attacks on the U.S. were shocking but the attack on Pearl Harbor—which now seems almost obvious given the amount of time it took Japan’s air force carriers to get into position—was just as shocking. The U.S. had one weak link in protecting the Pacific bases, he said, which Japan was able to manipulate to its advantage. “We have to be aware of the weakest things we do as civilization,” he said.

Survey the data landscape. “The first step in a prediction is a more careful evaluation of the past,” he said. Most fiscal models are built on making inferences from past data, and good recordkeeping is important to knowing where you’ll be in the future.

Try and err. The important thing to remember, Silver said, is that there is a learning curve with complex systems. Most successful companies get the notion that it’s important to make small gains around the margin. “You’re looking for small wins over time instead of a big miracle,” he said.