Glad to see that Jakob Nielsen has recently restated his position about how many users you need to test with in usability studies - the answer being 5.
We have been arguing this for years but still find we have to justify to clients why such a small number is acceptable and quite often do more than this just to meet the needs of our sponsor's requirement that the study has credibility with the people they report to. As a result they end up paying more for study than they need to.
Key points that are worth restating are:
- Usability studies are primarily about observing behaviours, while market research studies look at attitudes. You need larger sample sizes to be sure you have accurately understood users attitudes but 5 users will identify 85% of usability issues.
- You are not relying on just the research evidence to provide insights. The observers draw on their own knowledge of usability issues when interpreting users' behaviours. Often you will watch somebody do something, which while not anticipated is very logical, which highlights an issue. In this case 1 user will provide the required insight.
- While the research by Tom Landauer showed that you need 15 users to identify all the usability issues with a site, 5 testers identifies 85% of usability issues. As Nielsen points out in his 2000 article it is better and more cost effective to retest to ensure issues have been fixed (because often they haven't been) then to blow the budget on one large test.
Nielsen also lists weak arguments for testing with more users, one of which had particular resonance for me because it had just been put to me by a client who said: "Our website has millions of visitors testing it with 5 users won't be enough"
Clearly this is irrelevant. As Nielsen says "An opinion poll needs the same number of respondents to find out who will be elected mayor of Pittsburgh or president of France. The variance in statistical sampling is determined by the sample size, not the size of the full population from which the sample was drawn". Basic statistics!
He goes on to say "In user testing, we focus on a website's functionality to see which design elements are easy or difficult to use. The evaluation of a design element's quality is independent of how many people use it". Too true. Clients take note. But then if you want to spend more money than you need to!
