Hi, this is Jing, a data scientist with great passion for applying data science and big data technology in the industry. In my past few years, I have designed, implemented, and analysed a lot of web experiments (we call AB test in following text). More than half of them failed or inconclusive, but all of them brought good insights to the team in terms of how to deliver a better product to our customers. The key to have a good idea to test is to have a lot of ideas. To be able to test an idea, a hypothesis is needed for starting an A/B test. In today’s blog I want to talk about hypotheses and prioritisation when you have a lot of ideas!
Translate ideas into hypothesis
Someone said before, opinions are like assholes, everyone has one. Not everyone could tell the difference between an idea and an opinion. Helping your colleagues or stakeholders translate their “ideas” into hypothesis is a process filtering their personal opinions. Since we can not run a test on someone opinion or gut feelings, it should be some evidence supporting the idea what we need to change and what we could expect.
If you are developing a product, it can be common that so many people around you eagerly telling you what you need to do to improve the product. People are so confident about their ‘ideas’ and give you a feeling that they know the product better than you. Honestly, the enthusiasm for sharing is really appreciated and we need to have a lot of ideas. But in reality it can be exhausting to deal with all these ‘ideas’. So I suggest we could have a ‘idea repository’ which is open to everyone so that people could put their ‘ideas’ there for product improvement. And your product team will go through it from time to time, evaluate it and prioritise on the valuable ones. However, there is a requirement on how to write on their ideas. The idea should be translated into a test hypothesis! It is not that hard, and everyone could be able to do that with some help from analysts or anyone who know how to do it! But it helps you filter out a lot of personal opinions and meanwhile gather a lot of good test ideas!
Good formula for writing a hypothesis
A Hypothesis for an experiment, you should brief it into a concise sentence so that people can quickly get what you are trying to test, why you do this test and what is expected.
For why we need to run this test
Since we have observed that… XXX (quantitative/qualitative data, previous A/B-test, competitor analysis or best practice)
Don’t explain everything in the hypothesis. You can add the links to those research in the ‘Insight’ part in the test specification. If you are wondering what is a test specification, go to my another blog Proposal of AB test process within Product Teams
By doing … XXX (the change we want to make for the test variants)
You could be more specific in the ‘Action’ part in the test specification if necessary.
We could expect … XXX (increase/decrease primary metric)
XXX are the success metrics for this test. Success metrics should be set before running the test, not vice versa. Still, you can be more specific in the ‘Expected results’ in the test specification.
All the details about the test should be in the test specification, not in the test hypothesis. The test hypothesis should be simple and concise.
Prioritisation
When we have a repository of test hypothesis, how we should prioritise on them comes into question. Here is a suggestion on how to prioritise by giving the hypothesis a priority score in terms of impact, confidence and ease.
Priority Score on the Average over (ICE score)
- Impact
- The expectation about the degree to which the idea will improve the metric being focused on, to our case is online conversion rate
- 1-10, higher number indicates higher impact
- Confidence
- The measurement of how strongly the idea generator believes the idea will produce the expected impact.
- 1-10, higher number indicates higher confidence
- Ease
- The measurement of the time and resources needed to run the experiment, e.g the traffic size on the page, the tech development complicity and etc…
- 1-10, higher number indicates easier experiment process
End
What I have written in this blog is not rocket science, but some suggestion and refections come from my experience in production optimisation. Hope this might be helpful to you. Thanks for reading. I am Jing, a data scientist aiming to be better and better.
JingsBlog 