Hi, this is Jing, a data scientist with great passion for applying data science and big data technology in the industry. In my past few years, I have designed, implemented, and analysed a lot of web experiments (we call A/B test in following text) in past years. More than half of them failed or inconclusive, but most of them brought good insights to the product teams in terms of how to deliver a better product to our customers. In this blog, I am gonna to talk about what is A/B test, why we need to do an A/B test and how the A/B test works from a technical perspective.
Concept of A/B Test (Web Experiment )
The A/B test is also known as split testing, is a method of comparing two versions of a web page or app against each other to determine which one performs better. Sometimes, there are more than two versions, it will be also called MVT (multi-variant testing). Properly speaking, A/B test is one type of web experiments and there are other types experiments, such as split url test, multipage test and etc.. depends on different categorisation. But usually we talking about A/B test as a reference to web experiments in general.
A/B Test
Use definition in Google Optimise, A/B Test is an experiment in which two variations of the same page are tested to know which performed better in providing the best user experience. Here, Variation A is the original version, and Variation B is a variant of that page, with one or more elements modified. If there are multiple elements in Variant B different from A, you are unable to tell which element brings the difference or the compound effect. So, it is better to run a Multivariant test in such cases.
Multivariate Test
Within a multivariate test, the variants include multiple combinations of elements to help find out which one creates more engagement. If there is one element has been changed, then it will be an AB test with two variants in total. If there are X elements has been changed, then it will be 2x variants in this experiment. Thus it becomes a multivariant test. Recommend this for smaller changes on one page such as content copy changes. So the compound effect by these smaller changes could show a statistical significant improvement on your product. However, if there are too many changes on the test variant, or the test variant is almost a different version from the control one, split URL test is recommended.
Split URL Test
In this type of test, the two different web pages compete against each other, and variations are defined by the URLs instead of by modified page elements. This type of test is often used when the variants are two different landing pages with a full page redesign. For this type of test, SEO might be a thing you need to take care. A friendly reminder here.
Why we need to run A/B test?
The simplest answer for why we need to run web experiment or doing AB test is because we want to optimise our online product but we are not sure whether we are on the right track or not.
Conversion Rate Optimisation (CRO)
What we are trying to optimise through A/B test is conversion when online users are using our product. You got certain amount of traffic or users for your product either from organic search or an aggressive campaign which your product team cannot really control. But what you can do is to maximise the the outcome for the traffic you got, we call it Conversion Rate Optimisation (CRO). Conversion is about getting your visitors to do what you want them to do in your product. It not necessarily means buying something, even though in E-commerce business conversion usually means selling a commodity. However, AB test does not limited to E-commerce. As long as you want to optimise your conversion rate, you could alway test it!
However, CRO does not equal to A/B test. AB test is just part of the CRO process. People talking about A/B test and they actually mean CRO. The main reason is A/B test is the core of CRO. CRO is a process, and also is a wheel. You need to make the wheel keep running and in the end, you achieve a better product. It is pretty unrealistic expectation on running an AB test and you will have a winner. But if you do the test properly, you definitely will learn something. So, besides A/B test, there are other 4 steps in a CRO wheel. Make sure you do each step properly.
- Create a solid Hypothesis from ideas
- Create a New design based on the hypothesis
- Experiment Implementation (including pre-testing, choosing the right testing tool…)
- Conduct Post-Test Analysis
- Following Actions based on learning
There are mainly 5 steps in the CRO optimization wheel, and different roles with the product team are involved. For step1, Find how to write hypothesis at How to A/B test different ideas on your website?. While UX designers are working on the step2, as an analyst you could do a Pre-test Analysis before turning on the AB Test. If you are trying to implement a test with some free test tools, check Quick Guide to use Google Optimize to run an AB Test! If you are planning to run multiple tests at the same time, check the Proposal of AB test process within Product Teams to read about how to make this wheel keep running with less chaos.
Time and Resource Efficiency
When it comes to how to optimize product for better conversion rate, team players within the product team can have different views. UX designers have a view on how we provide a good user experience, visually and sensually. PMs has a good plan on how the product should be now and will be in the future. Developers are aimed to deliver a product technically they will be proud of. Data Analysts are inspired by data on what the customers need and what our product is currently missing. Basically, everyone in the product team has an idea on how we should optimize our product from different perspectives. These ideas can be same, but usually, they can be so different from each other. It is good that we have so many ideas and everyone cares about the product by voicing their thoughts. Then comes to the question, who we should listen to? PM or UX? But we need to be data driven, then analyst? Developers says does these people actually know how the product is built? If we usually go into a debate like that, no one in the team will be happy and feel motivated. We don’t want that.
From my experience, “test it” will be the easiest way to solve the conflict and move on. Let the users decide on what suits them the most. They are the ones we should listen to. You might say, running a test can cost a lot of time and resources. But when your team get used to it, and reached to a consensus on this when it comes conflict ideas. Running a test can be very simple and time, resource efficient. Having meetings debating with others can be very time and resource exhausting! Last but not least, these optimization ideas are often very profitable and can be very scientific and data driven. In the end, stakeholders and team players are all convinced and agreed. If not, let run another iteration!
How does the A/B test work technically?
From technical perspective, there are only two types of A/B test (web experiment), client side testing and server side testing. The basic difference is where the variant is delivered. Why this is important to know, is how it technically works decides the limitation of use cases. It helps you to know which is the best tool to run your A/B test. Considering this post is already long, to read more details at A/B test: client side vs. server side.
Client Side Testing
Client side testing is the much more common than server side testing. The test variant is delivered at client side, that is users’ devices (mobile, tablet, desktop). However, this does not include APPs. Testing in APPs are done at server side. For most of AB test tool on the market, they provide client side testing.
Server Side Testing
Server side testing is not something analysts or PMs could run by themselves. Because the test variant is delivered in the web server, which means there are testing codes in your product server application. You need a developer to help you to implement a test. As a result it is not that common comparing to client side. You will say it is complicated, why not just test in client side. Things exist for a reason. Server side is complicated, but it also powerful. You could test really complicated cases. I will explain this in another post.
End
I will not say this is a very technical post. It is more like an introduction to A/B test. If you just started thinking about running A/B testing, I hope you will find this post useful. I will write more in this topic, A/B testing within product teams and try to keep each post short. Thanks for reading. I am Jing, a data scientist aiming to be better and better.
JingsBlog 