What is experimentation?
Experimentation delivers data for decision-making. Experimentation replaces decision-making based on intuition, or can also complement data-science projects.
In any situation where a decision needs to be made, one action should be taken with one group of people and another action with another group of people. Let me try to illustrate this with an example: say you are interested in knowing how to best communicate with your customers, or how to personalise your messaging – no one really knows the answer. How would anyone know? Everyone needs to figure it out, in this case, you could use an experiment for that. You divide your target audience into different groups, ideally at random, and then you expose one group to one ad for example, and another group to another ad. If one of the ads performs better than the other one in terms of the actions you want your customers to take, you have yourself a winner.
What should companies experiment with? – in short, anything but in reference to marketing or advertising then I can suggest the following;
Marketing, Customer Acquisition: Acquiring customers via different media channels, asking, which media channels are most cost-effective, or which media mix is.
Marketing Creatives: to attract the most relevant customers, often those that are most valuable customers.
Personalisation is one of the ultimate objectives in Marketing, and I was more and more often asked of how one could personalise the communication with their customers
Why and what type of companies should experiment?
In short: all companies! It is rather rare to be perfect from the get go, and no matter where you start, experimentation helps you reach your goals in a very systematic and data-driven manner.
How should you experiment?
Know what you want to know (ideally keep it simple): Formulate Testable Hypothesis! A null hypothesis and an alternative hypothesis are mutually exclusive. If one is true, the other one is false. And on top of that, it has to be measurable. The decision criteria to find or not to find a difference between treatment and control is what should show up in both the hypotheses.
Develop a testing framework: A testing framework is about RnR, prioritisation of tests you want to run, what can be tested with alternative methods (f ex causal impact), but also how to make decisions if a test is a success or a failure (measurement framework). Every experiment comes at a cost, the direct cost of running it with all the resources necessary, but also an opportunity cost as an Economist would say: what would have been the value of not having run the experiment. That requires, of course, a framework allowing to prioritise hypotheses for testing.
Act upon the results, even if you don’t like them. Could be that you are ready to implement a new feature, maybe the test raised another question you would like to investigate, or maybe the test clearly indicated to drop whatever idea was up for the test. Act on the result.
Create a culture around experimentation: if you run experiments, but do not act upon the results, your employees will ask themselves why they should use their time on experimentation at all. Encourage decision-making based on data rather than intuition. Or put your intuitions to the test, if your intuition was wrong, own it, celebrate failures in experimentation and embrace them as a learning. And most importantly: don’t rest on your knowledge you acquired one day, continue experimenting, never stop learning.
There you have it – Experimentation replaces decision-making based on intuition, or can also complement data-science projects.
