What We Learned Last Night About Data & Prediction Is Surprising

Last night, data and prediction lost in the most profound and significant way I can remember.

For those of us in the business of performance marketing (meaning we market in a way where we measure everything people do and nothing really that they say they are going to do), last night was—well, honestly, it was still shocking. I have a couple of opinions as to why. No doubt people will be parsing various elements of the electorate—why this population or that did or did not vote the way the polls predicted. I don’t want to talk about the candidates or the policies, I want to talk about the data, and the incredible failure of the polls to predict and the populace to imagine.

The Polls Did Not Predict

Are our polls biased? Are our models broken? Was there something sinister going on? I don’t think so. I think this is something else. It is something that we’ve seen time and time again when we poll people or do research on products or messages. It is not about biases or questions or models. It is plain and simple that people do not know what they are going to do ahead of time.

It is impossible to predict what we will do. Predicting what everyone will do? Forget about it. The only measure that matters is behavior. Watch what people do and you will know what they will do. Ask them what they will do, and you set yourself up to be stunned. Bigly.

In 2006, Nokia did a worldwide study on the mobile phone market and found conclusively that there was not a market for a smartphone. They released their findings six months before the launch of the iPhone, and anyone under 25 reading this is thinking Noki-who? Turns out people did want a smartphone, they just couldn’t imagine it.

It’s Hard To Collectively Imagine

Polling the collective imagination. There’s a business opportunity. The problem is that asking people what they are going to do or what they want is a tough business. You can find study after study that points to this fact, but we keep on asking and then believing the results as though if we just ask the question the right way, then we’ll be able to divine the answer.

The innovation curve will tell you that there are relatively few innovators, some early adopters, an early majority and that most people are late adopters or laggards. This means that there are a few people imagining and then risking to build (or destroy) the future and the rest of people getting surprised by what they do. We cannot poll the laggards and ask them to suddenly invent a future. It’s just not going to work out. But that doesn’t stop us from asking.

The world changes more from discontinuities than from the slow wearing of time or the incremental steps of everyone. It is changed by the things that Nassim Nicholas Taleb calls the ‘black swans’—those possibilities that we don’t even consider or cannot collectively imagine. Fortunes are made and lost on these days. Nations are changed on these days: assassinations and elections, inventions and wars, booms and busts, scandals and triumphs.

There’s not a reliable statistical model for these discontinuities. There are probabilities, and when we do not spend the time to imagine our responses to these unlikely events—when we forget that even at one in a million the math is “telling us there’s a chance”—then we get caught flat footed, mouth agape, deer in the headlights. This was both a polling failure and a failure by us to realize what those polls were telling us.

Regardless of how you feel about the outcome there are some lessons from it all: Precious little is 100% certain. Behavior is unpredictable. Probability is not certainty. Even the unlikely is still possible.