The truth hurts when you ask the wrong question

Bryan Melmed

When second guessed Deep Thought responds:

"I checked it very thoroughly," said the computer, "and that quite definitely is the answer. I think the problem, to be quite honest with you, is that you've never actually known what the question is."

Today's marketers are embracing increasingly sophisticated findings driven by massive amounts of processing power and terabytes of data. Like Deep Thought, our systems are happy to help us however they can - without questioning our intentions. Starting with the right question is critical. The real danger is not an indecipherable answer but one that seems plausible enough to work with, or worse, sets an artificial and unrepresentative framework for measuring success.

Mind the gap

One way to understand this problem is by identifying the gap between how we word the question in our head and how computers are able to interpret it. For example, we often wonder "Who is my best customer?" The concept seems so clear at first, only to fall apart when a computer demands a specific measure of 'best'. Our query devolves to something easily measured - even as far as "who is clicking my ad?" which is hardly the best customer at all, and yet so often used as a measure of success.

Even a standard lookalike model brings with it a gap between your real question and the answer provided. "How can I find my next customer" is implemented as "Find someone who looks exactly like my last customer" - ground so familiar to tread that we almost forget that the real question was "Who should I be serving ads to?" And that's typically not the best customer, but the one that needs a little push. Advertising to users on their way to purchase will look widely successful when measured on last-click or last-view attribution, but is still an enormous waste. You should be serving ads to the people who are most likely to be influenced, not the ones who are already convinced, and not the ones who will never be.

Ace the base

I was brought into a sporting equipment chain a few years ago that were struggling to improve their marketing efforts. It soon became clear that they had what I call a baseline problem. Against a general population, their systems had identified the "best" customers as athletic types - the weekend warriors who were spending thousands of dollars on equipment. Yet this audience needed no reminder that the stores existed, and even felt resentful that the chain had huge market share and pricing power. Advertising to this group had literally backfired.

The true baseline was a population they could influence - at a minimum, consumers of sporting goods. When comparing customers against a more representative population, the computers had a different answer: target the casual sporting goods buyer. Families buying equipment for kids, and beginners needing advice on how to get started, had smaller but more frequent purchases. Moreover, the population was huge. To truly connect with the right audience, the company just had to base the question on a more representative baseline.

A similar problem emerged when I was working with a digital agency for a well-known pizza chain. Here the goal was to encourage online orders, and their advertising had simply targeted visitors to the website who left without making a purchase. Again, the baseline was wrong. Some of these users would never put in their order online - for example, teenagers who didn't have credit cards. The true baseline was not all visitors, but those who were making online purchases elsewhere. Against this population, we could see that some of these site abandoners were busy mums who were unfamiliar with the ordering process. Their phone orders now arrive with a unique customer code and a discount for their next online purchase. That little push is all it takes to move the orders online, where they inevitably remain.

Get the question right and you may be surprised by what you achieve. Frances Bacon once said that "A prudish question is one-half of wisdom." As applied to big data insights, I'd say the percentage is much higher.