“When a measure becomes a target, it ceases to be a good measure.”
— Goodheart’s Law*
“We often think of tech as a net positive but without strong discernment, relying on data flows and their source algorithms can result in wasted time and effort—blinding us to potential opportunities and reinforcing habitual patterns of action and thinking that may no longer serve.”
If we are very interested in setting and meeting KPIs and/or identifying an ROI we have become tilted toward an outcome or result, such as a cost per conversion or number of sales within a period.
At MSB, we try (with varied success) to not be results oriented. We try instead to support processes that allow for the exploration of as broad a swath of the outcome space as possible—leading to improved results over time. This isn’t easy, the internal and external resistance to let go of “measurable results” is understandable—especially when the offered alternative is necessarily unclear.
A narrow definition of success means variance from the metric a). produces no insight and b). looks like failure. Paraphrasing Goodheart, if the metric is the target, then the results are potentially meaningless. At the very least we will be unable to develop a robust evaluation of what happened.
Let me share an example. In early 2020, we were approached by a client who had a purchase order (PO) set aside by a major retailer (that had just closed their stores nationwide, due to Covid). Faced with a significant shortfall, they asked us to help them go direct to the consumer. The request, sell 900 units in the following 90 days.
This was a new brand and a new product so, with all possible caveats in place, we set up an Ads funnel driving traffic to their website.
One small detail: the client did not remove the link to the national retailer’s website from their homepage.
As a result of the click through traffic generated by our campaign to the retailer’s website—and the subsequent pressure that potential customers put on that retailer—the original PO was fulfilled by the retailer within 23 days of our start.
A win! Right?
Looked at from a purely D2C play, it was a failure—we only sold a few dozen units. But the unexpected outcome was a saving grace and while our efforts were anecdotally acknowledged in that outcome, our unit sales stats were, in the end, meaningless.
What if things had gone the other way? What if the purchasing process at the retailer was unwieldy and slow? What if they were risk-averse and chose not to fulfill the PO?
We were focused on only one outcome, we were only measuring one result—onsite sales. Had we measured the click-throughs to the retailer, from the beginning, what else would we have been able to do? If our scope of possible outcomes was even broader, what else could we have measured—and how would that have impacted our actions?
If you’re focused on specific outcomes you necessarily limit what can be learned. Both on the vertical axis from good to bad (0—1) and also on the horizontal axis in terms of capturing the potential of other things that might occur. When focused on a specific outcome, essentially, the horizontal axis doesn’t exist—there’s just an up and down toggle, hit the goal=good, missed the goal=bad.
It’s not that other things, other outcomes, won’t occur, just that the benefits or detriments of those outcomes won’t be measured and will therefore be invisible to any evaluation.
April 2022
* Goodheart’s Law, is named after British economist Charles Goodhart, who advanced the idea in a 1975 article on monetary policy in the United Kingdom.