We sometimes come across clients who argue that they don’t need to do any UX testing because they undertake A/B or multivariate testing. They say that these approaches provide hard evidence of what actually works. It is hard to argue with “We made this change and there was a 10% uplift in sales”.
However, A/B testing and UX research are not competing approaches - they are complementary. In order to identify a change that you can A/B test you need to be aware of a problem. This understanding may come from an analysis of site analytics or other ways, but UX research is sure-fire way of identifying issues and understanding the ‘why’ as well as the ‘what’. A typical UX session we run will usually identify between 50 and 100 issues that affect the user experience in one way or another, many of which the client is unaware of. That is a lot of new things that could be subject to A/B testing.
Even after a problem has been identified a solution needs to be developed. A/B testing a poor solution may erroneously lead to the conclusion that what was thought to be a problem in fact wasn’t. We are often presented with page variants to test where testers struggle to notice the difference between them. What seems like interesting differences to the designer are simply not noticed by testers. UX research can provide good pointers as to what sorts of solutions might work best and, therefore, improve the effectiveness of the A/B testing.
What UX research does not do well is say which solution (assuming it is well designed) is the best – that is where A/B testing should come into its own. However, even then there are circumstances where UX research is best. A/B testing requires that you have a way of measuring the success of the variants you are testing. This may be straightforward for an ecommerce site where conversion rates can be monitored, but less straightforward to an information site. How do you measure the success of two ways of presenting information about cancer to cancer sufferers? So in these cases UX research may still be the best way to get the insights required.
So to summarise, conduct UX research first to identify what the issues are then A/B test your solutions to discover which works best!
