A/B test: client side vs. server side.

Hi, this is Jing, a data scientist with great passion for applying data science and big data technology in the industry. In my past few years, I have designed, implemented, and analyzed a lot of web experiments (we call A/B test in following text) in past years. In the beginning, I didn’t pay much attention to how A/B test actually technically works as long as they meet my requirement listed below. As a result, our talented developers just built a simple traffic allocator for running A/B test in our server side. It worked and works really well in fact, but it slows down the product development pace at the same time since it took too much dev resources. Then I begin to look into how to run A/B test ‘properly ‘ .

• Tested users are randomly allocated into different test groups
• The test user should always be allocated to the same test group during the test period to ensure consistent experience
• There should be certain tracking in terms of who has been tested in which test and in which test group from data perspective.

How does the A/B test work technically?

From technical perspective, there are only two types of A/B test (web experiment), client side testing and server side testing. The basic difference is where the variant is delivered.

Client Side Testing

Client side testing is the much more common than server side testing. The test variant is delivered at client side which is users’ devices (mobile, tablet, desktop). Almost all the test tools on the market provide client side test. Many of them include a visual editor which free you from touching the code while running the test.

Use cases with Visual Editor

Of course there is a limitation on what type of use cases you could run the test by yourself in the visual editor. The users case basically are visual changes like:

• Copy changes on the existing component on the web page
• CTA link change
  • Text of the CTA or the destination of the CTA
• Position of an existing component
  • Change the order of the component or remove certain component
• Change an image/ styling of an existing component
• A new simple component just with images, links or texts (no functions)

This does not mean, client side test can only do these type of use cases. Without visual editor, a new module or a totally new feature which complicated functions can also be also tested through client side by developers. If developers are involved, then usually server side test is more recommended since there is an common issue to all the client side test, Flicker issue.

Flickering Issue

Flickering is when original content appears to a user before an experimental version showing during testing. It is a common problem for most testing tool when implementing a client side test. This happens because of an asynchronous script with the test tool. Meanwhile, this asynchronous makes the test page not be slowed down due to the test.

In the Client Side Testing Figure, the original version is loaded first on user’s device, then the test version is loaded again to cover the original version. However, the flickering only becomes a problem when users can actually see it. That means, for the test whose changes are below the fold, flickering will not be a problem since user will not be able to notice the flickering. So, just do the test. When it comes to the changes above the fold, Hero banner or navigator change, server side test will be a better solution.

Other limitations

Since client side test rendering the test variant at users’s device side, cross channel or devices is hardly possible. Some test variant which is not compatible with the browser might pollute the result data. Besides, testing in Mobile APPs are usually done at server side since apps serves dynamic content.

Server Side Testing

Server side testing is very powerful supporting for most of the use cases which needs to be done by developers. Because the test variant is delivered in the web server, which means testing code is in your product server. A developer is needed to implement a test.

No limitation on Use cases

I will say there is almost no limitations on the use case for server side test. If we have enough dev resources, we could run all the test on the server side for better control, cross devices, channels and etc… There are some typical use cases which client side test can not support

• Test performance for different algorithms such as recommendation
• A different flow which multiple features or pages are involved
  • a new sign-up flow or a new shopping flow
• Targeted users cross different channels and devices
  • Targeting on certain group of logged in users ( not anonymous to our server)

Caching

If you use the server side test without thinking about caching, then you might run into an “uneven split” situation, which means, a lot of returning online visitors are keep getting the control version while they are supposed to be in the test group. This is due to the cache. Since usually there is a content delivery network (CDN) between your web server and your end visitors. For the returning visitors, their online content request will only reach to CDN, and content for them will be delivered from CDN cache. They will not reach to the web server where decide which test group they will be. Thus making the integration of CDN is compulsory for running server-side test. Otherwise, only the first time online visitors will be tested.

End

I will keep update this one when I know more about this two types of tests. People have questioned about why an analyst need to know these technical staff. My answer is, I am a data scientist, not a pure data analyst or digital analyst. I want to know how things work fundamentally and where my data comes from. What’s more important, I want to bridge the knowledge gap between the dev and the product owners/managers to fasten product optimization process, by finding the suitable solution and choosing the best tool. That is what I think a data scientist should do when he/she is supporting a product team. Data scientists are never limited to build models. The core about being a data scientist is ‘cross-disciplinary’, you know data, but not just data. That’s my personal opinion. If I offended anyone, sorry for that. That was never my intention. And thanks for reading. I am Jing, a data scientist aiming to be better and better.

Advertisement