Recently, I worked on a few advertising effectiveness studies where, despite failing to achieve statistically significant differences in conversion rates between exposed and unexposed, expected profit return was highly positive. So much so that any CFO or portfolio manager would make the investment in a heartbeat.

Yet analysts were reluctant to report that the tests were successful.

This disconnect between statistical significance and economic significance can easily occur for high end offerings where conversion rates are typically small…1% or lower…but where the profit value of a single conversion is really high (think boats, solar panel installations).

When the two are at odds, I propose that business decision-making should be based on ECONOMIC SIGNIFICANCE* first and foremost. Stat testing math can easily lead management in the wrong direction.

What do I mean by “Economic significance”? It is the comparison of the expected profit return on the investment relative to the cost (e.g. campaign ad budget). High economic significance can happen regardless of p-values.

How bad is the problem?

For high-profit items (boats, solar panels, cable TV subscriptions, insurance, etc.) you are more likely to encounter this paradox…conversion percentages are not statistically different, yet the expected value of the profit function can be 10+ times bigger than the media cost investment.

The disconnect can also exist at the opposite end of the spectrum. For CPG products like facial tissue or orange juice, you are much more likely to get a statistically significant result, but it might imply a profit loss rather than gain.

So, what I am trying to convey is that statistical significance and economic significance do not go hand in hand…in fact, they can go in opposite directions and when they do, marketers should follow the economic significance calculation.

How can economic significance be calculated?

For recent studies on high-profit items, I generated 1,000 Monte Carlo simulations of exposed vs. unexposed results. You can use either binomial or Gamma distributions but I like the latter because with really low conversion rates, the assumption that everything approximates a symmetric normal bell curve is not very good. The profit function for high profit offerings is actually highly asymmetric which the gamma distribution better reflects demonstrating that there is a lot more upside than downside.

For each simulation, calculate the expected profit return and average those results. You might cap negative lift at 0 (I would except for a few well known marketing gaffs).

However, I find that even for statistically insignificant differences…say, where up to 35% of the simulations show zero or negative lift…the penalty for failure is much smaller than the upside of success. Taking an expected value, the net result can be a highly positive expected ROI even though the test failed to meet 90% confidence.

What would I want to know as a decision maker?

Two things really:

1. What is the chance of failure — i.e., losing my ad investment without incremental sales offset?

2. What is the expected return relative to the ad budget?

With those two pieces of information, I can make an informed decision. On top of that, statistical significance on a percentage lift adds little as a deciding factor.

What about brand lift measurement?

The problem with survey-based brand metrics is that marketers usually have no idea what a point of awareness or positive purchase intent is worth financially, so already, they are doing research divorced from economic significance measurement. This was actually the core problem that the MMA solved when it created its “brand as performance” (BaP) study protocols (disclaimer: I was the mad scientist behind this). In that single source research (ad serving, conversion data, and brand perceptions all from the same IDs) we actually DO quantify the value of brand favorability in terms of sales annuities over time. We didn’t think to call it “economic significance measurement” but that is what it is.

Do we do stat testing with BaP studies? Yes, but always we also provide an economic significance analysis that talk both CMO and CFO language.

Final thought: For ad testing and advertising decisions, there should be nothing sacrosanct about stat testing. In fact, it can get in the way of good decision-making. So please do the due diligence of going through an expected profit return calculation. You might find that advertising is a much better bet than you thought it was.

Endnote: Does stat testing have any role?

Definitely. If you are doing product testing, you need to be 99%+ sure the product change will cause no harm. If you are doing claim substantiation research, statistical significance is essential. In regression models, sometimes you want to drop out one predictor variable from a pair that are highly intercorrelated so you drop the one with the poorer p value.

*Credit: I owe the phrase “Economic significance” to Prof. Koen Pauwels. Great insight and a great phrase.