I'm pushing out a new feature soon that has a lot of different UX components. The key metric that I want to track is user engagement (I have the equation for what this means for me and how to measure it). What I want to measure is the best combination of the UX components that maximizes the engagement. The problem is that there are a lot of variations in terms of what UX components I can display at any time, and how I can display them.

To provide some examples, consider a widget with tiles where each tile represents a book (Note: this is just a simple example, my actual product has more variations). Clicking on one of the book tiles opens up more details about the book but thats irrelevant. What I want to maximize is the chance that the user clicks on one of the tiles. Each tile can have a thumbnail, the name of the author, rating from Goodreads and the year in which it was written etc. Each one of the components (Author Name, Rating and Year) can be hidden or displayed.

The first answer that comes to my mind is AB testing. For example, Week 1, deploy AB test two versions of the widget. One that only has thumbnails alone and the other that thumbnails and Goodreads rating. Week 2 will depend on the results of week 1, if week 1 gives us the result that rating gets more engagement, then the new test will be thumbnails + ratings vs. thumbnails + ratings + author name. If week 1 gave us that ratings gets less engagement, then we can repeat the same test, but this time replacing ratings with author name. And we will can follow the similar approach for week 3 and so on.

My concern is that this process seems a little inefficient. For weeks, I will be just experimenting with different variations of the widget and drawing conclusions. Data of each phase will be limited to the length of the experiment. Who know how long it would take for me to find an ideal combinations.

Given my lack of experience when it comes to solving problems like this, I'm a little hesitant to move forward. My questions are

  • Is AB testing even the right solution to this kind of a problem?
  • If yes, then is my approach (week 1 do x, week 2 do y etc) correct when it comes to the implementation?
  • What improvements can I make to this process? Would be great if you can point me to some helpful resources regarding this topic.
  • 1
    Questions containing words like "right" and "correct" are difficult to answer specifically. You've already expressed concern about how long this process will take: I don't think your concern is misplaced. Have you considered setting up some focus groups? – Robert Harvey Apr 9 '18 at 22:40

Is AB testing even the right solution to this kind of a problem?

Like amon said in his post: A/B tests can be a powerful tool to test a hypothesis but there are also other ways.

When you're talking about fine-grained optimisation, you might want to take a data-driven approach like A/B testing, but just asking end-users is often easily overlooked. In software engineering it is generally very valuable to hear end-users' feedback. Like this, there are more, simpler ways to test the effectiveness of features.

If yes, then is my approach (week 1 do x, week 2 do y etc) correct when it comes to the implementation?

This works, but you have to take into account the noise that is created by other factors that play in week 1 and week 2. Also note that this approach will also take n weeks, where n = amount of features.

What improvements can I make to this process? Would be great if you can point me to some helpful resources regarding this topic.

Ideally you would want to test multiple versions of a feature in parallel, each version tested against a percentage of the user-base. However this requires your production infrastructure to allow for this kind of releasing.


A/B tests can be a powerful tool to test a hypothesis about your user interface. A hypothesis is a falsifiable proposition, e.g. “including a rating in each book tile increases engagement”. You can then implement an experiment and put your hypothesis to the test. As this is a statistical experiment, you need to decide on certain parameters such as confidence intervals for the rejection/acceptance of the hypothesis, sampling method, and sampling size. Once the experiment has completed, you can perform an analysis and get one of three results: the hypothesis was accepted, rejected, or is inconclusive.

You suggested a sampling method that runs each variant consecutively for a week. This is not a very good sampling method, because you have added another parameter “time” to the experiment. Maybe the engagement rate is time-dependent? With such an experiment design you cannot tell. Also, some users may revisit the site during the experiment and experience both variants. The order in which the variants are presented may be relevant, and adds another dimension to the experiment which you did not consider in the design – you might get different results depending on whether you run A or B first.

To avoid such sampling problems: first calculate the sample size that you need for your desired statistical power. If your expected effects are small, you will need larger samples to show a conclusive result. When you compare the necessary sample size with your expected number of users, you can find out how long you have to run your experiment for. Ideally, run the experiment for an integer number of weeks so that the effects of weekdays can be eliminated (especially weekdays vs. weekends). During the experiment you can then randomly sort your users 50/50 into the A/B groups, and keep them in that group for the duration of the experiment.

For some changes you don't want to expose half your users to a new idea, just to take it away later. If you have enough users, it might make sense to run the experiment on a fraction of your users, or to run a pilot study before the actual experiment with a tiny sample of users.

Even when you get a conclusive result, your user base may not be uniform. A change that improves your user interface on average may still worsen it for some. So check for that first – a confirmed hypothesis is not a “go” decision!

All of that is a lot of effort. Designing a good experiment is hard. Getting a good sample might take a long time. The experiment may have ethical or legal implications that must be cleared first. The statistics might get non-trivial.

This effort is often not worth it. Once you have a steady stream of users and have implemented the necessary A/B test infrastructure, being able to have this data is nice! But before that, they might be a massive waste of your time.

There are many simpler ways to get feedback on your user interface.

  • Trust your guts. A lot of design is common sense and/or doesn't matter. This is how you'll handle most decisions anyway.

  • Ask other people for quick feedback, but ask a useful question. E.g. “Do you think showing a rating like this will make fiction readers more curios about the book?” (Though that borders on a leading question.)

  • Create personas. A persona is a fictional model user with a set of goals and desires. Validate your design by checking whether this design helps the personas achieve their goal. Tip: include personas with limitations, e.g. colour blindness or slow internet connections.

  • Make feedback easy, then just make the change. If it was a bad change, users will complain. Although users will always complain about any change. Openly communicating about your intentions (e.g. blogging) can give you early feedback from power users.

  • Ask questionnaires. Such surveys are bad at producing statistically reliable data because of sampling problems, but can still produce valuable insights from your users. E.g. you could ask which factors make them interested in a book, and offer various suggestions (cover, author, blurb, average user rating, critic ratings, …). Designing good surveys is more difficult than designing an A/B experiment, but you can run a survey before having implemented the feature. E.g. you might find that few users are interested in user ratings, but some would really like to see a content description.

  • Use focus groups and interview people. This won't give you quantitative results like an A/B test, but can unearth valuable observations. One big advantage is that you can do interviews before you have a user base or even a functional product, which allows you to validate ideas early. You can also use interviews to refine your personas.

So yes, A/B tests are great! Just like Porsche sports cars are great. But sometimes all you need is a bicycle.

  • hey thanks for such a detailed answer. I personally agree with the most of what you said, however we live in a time where data driven decisions are celebrated and going with one's gut is ridiculed. In order for me to present an argument why X is better than Y, getting data is the easiest and fastest way. Questionnaires, surveys and peer feedback can paint an inaccurate picture depending on the people I have access to and their cognitive biases – satnam Apr 10 '18 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.