The holy grail of marketing is the ability to predict without error the future behavior of your customers.  Whether you want to know when they’ll buy a new widget or how much they’d be willing to pay, foreseeing the future is the goal of almost every web analytics package, psychographic profile report, and focus group.  We invest millions each year trying to predict consumer behavior and model public opinion.

The underlying question, though, is in effectiveness.  Will our new model accurately predict what we want it to?  How reliable will our new dataset be?  Can I really stake my career on the outcomes of this study?

One of the better-known systems for modeling is the rankings of collegiate football teams.  These rankings are made up of several dimensions: the raw data about each team (scores, margins, yards per game), objective comparisons with other teams (the same raw data in relation to the teams they’ve played), and subjective analysis from “experts” in the field.  The objective is to yield a ranking where team #1 is better than team #2 which is better than team #3 and so on.

On Saturday, my alma mater (Go Ducks!) went into their game ranked #10 in the nation.  Their opponent, USC, was ranked either #4 or #5, depending on which ranking you looked at.  According to either ranking system, though, USC should have won the game.  The reality?  My Ducks beat USC by more than 20 points.

While some people I know place all their faith in team rankings, this weekend proved how unreliable the system can be.  Now we have new data, and the “experts” will crunch new numbers and give us “better” rankings.  Will these rankings be more reliable than last week’s?  Not by a long shot.

Using statistics to predict behavior and outcomes is important to the science of marketing.  However, we can never let ourselves walk the naive path of blind acceptance.  No matter how much we trust our data or how impressive the results might be, we must always approach our models with a note of skepticism.  We could be wrong …