First off, huge congratulations to David Card, Joshua Angrist, and Guido Imbens, who were awarded the Nobel Prize in Economics this week.
They were awarded the prize based on their work with “Natural Experiments”, an innovative method of empirical research developed in the 1990s.
That’s great news, but what does this mean for experimentation and conversion rate optimization (CRO)?
For decades innovators have been relying on randomized controlled trials and observational studies to drive their respective industries forward. In recent years, there has been an emergence of more innovative techniques, and now a newly recognized method of natural experimentation.
Each tactic adds to a growing toolkit allowing researchers to make evidenced based decisions.
Observational Studies – The original tool
Observational studies are used by marketers to collect direct and indirect information about customers in natural and planned environments. It is used extensively in business to consumer marketing (B2C), driving the collection of behavioural data, measuring consumer behaviour directly and “watching” how the consumer behaves.
Observational studies are powerful tools, but can be limiting because they can not be used to prove causal relationships on their own. Without more structure and careful analysis, observational studies allow the interpreter to inject their own bias into their research.
Randomized Controlled Trials are the traditional form of an experiment. You control the parameters. A testable sample population is built based on variables an experimenter can control. Then, subjects are randomly assigned to two different groups, with an experimental group receiving a variation, and one control group receiving no variation. Results are then assessed to determine how effective the newly introduced variation was.
The randomized control trial
When conducting A/B testing, effective experimenters are able to control many of the variables found in the natural world (like time, and audience). This allows experimenters to test hypotheses and prove causality.
At Widerfunnel, we have always been proponents of the tremendous value of experimentation. With our client partners, we have found new ways to understand consumer behaviour and drive revenue with evidenced based decision making.
However, it is not always possible to control variables neatly in a randomized controlled trial.
What if we want to prove causality in large uncontrolled scenarios in the real world? For example, what if we wanted to understand the effects of education on future income (the key example cited in the Nobel Prize Report). That’s where natural experiments have largely been conducted, which have led to the Nobel Prize.
Natural Experiments – Evidenced based decision making in the real world
Natural experimentation occurs in real life situations where causal relationships are examined. In these “real world” scenarios, it is difficult to control many of the variables used when performing experiments, a historical hurdle when trying to prove hypotheses. Scientists of all disciplines have always known the power of evidenced based decision making, but without being able to control the parameters of a sample population, could run into obstacles when reaching a conclusion on statistical significance.
This week’s announcement provides a recognized framework for tackling such problems.
The solution? Natural experiments – situations arising in real life that resemble Randomized Controlled Trials. These natural experiments may be due to natural random variations, institutional rules or policy changes.
The winners were able to call out bad data, and reduce the room for error historically associated with natural experiments. Pretty incredible stuff right?
We thought so too. This week’s announcement was further recognition that evidenced based decision making provides the best results, and not just on websites. Natural experimenters now have a Nobel recognized methodology where they can use the same principles in their conclusions that we have provided to our partners since our founding.
Our mission at Widerfunnel is to help our partners drive evidenced based decisions through designing more insightful experimentation programs that maximize return on investment. Want to learn more about what we can do for your team? Reach out to us today.
On behalf of all digital marketers that run conversion rate optimization, experimentation, A/B testing efforts, we salute David Card, Joshua Angrist, and Guido Imbens for paving the way on the acceptance of natural experiments.
Subscribe to get experimentation insights straight to your inbox.