With thousands of apps on the market, users are overwhelmed with choice. To ensure an app is successful users need to be able to find it, think that it's relevant to them and find it easy to use.
In this article we argue that testing an app with users is vital to addressing these issues. We also set out some early stage outcomes from our app testing programme and the approach we have found best for testing apps.
User testing apps will help with:
According to Apple there are 225,000 iPhone apps in their App Store (June 2010). There are rapidly increas-ing numbers of apps for other platforms as well. Getting your app noticed is, therefore, quite a problem.
Most app stores rank apps in lists of top apps. These apps account for the vast majority of app downloads. Precisely how these are compiled is not usually revealed, but it seems it is largely down to recent downloads.
Typically when searching for an app, the position in the store search results is determined by the app name, the keywords in the app description, the price, the app popularity (downloads) and the ratings.
According to a recent Econsultancy report (August 2010), searching for a specific app, browsing through top app store rankings, and word-of-mouth recommendations are the three main ways that users find the apps they want to download. So, having a name and description for an app that fits with what users look for is very important in improving rankings, and having a useful and usable app will improve ratings. We look at the app selection process in our user test-ing to generate insights to help your app stand out from the crowd.
It seems obvious, but an app has to be 'useful' or 'entertaining'; there are thousands of apps which seem to be neither. For organisations, what matters is 'useful' (farting apps and Angry birds may be popular but are not very relevant to organisations!). Therefore, it is important to be sure about what users really want to do on the smart phones and pad; they don't necessarily want to do the same things as on their PCs or laptops. Apps should do no more than users want. Redundant content or functionality can get in the way of the things users really want to do. User research will identify precisely the goals users want to achieve and help develop highly focused apps.
Recent research by Jakob Nielsen (Alert Box May 2010) flags up that apps have usability issues because "anything you can show and touch can be a UI. There are no standards and no expectations. He goes on to say that as a result users' can't transfer their skills from one app to the next.
He quotes a nice example - touching a picture could produce any of the following 5 results:
What this flags up is that app developers are at liberty to design things pretty much as they like, without the same constraints that conventions and established good practice force on them when designing for the web.
If this design work is done without reference to users, then the potential to produce unusable apps is high. For example the 'Wine Snob' app has been designed by someone who clearly thought it would be a great idea to use wine colours for some text, background colours, icons etc. For anyone without 20:20 vision (and even for some of these) it is unreadable. So, clearly, checking with users that an app is usable is vital.
User testing allows you to check whether those great ideas of your designer really work for users.
Bad apps can destroy brand value. Organisations spend millions on building their brands and then often produce an app that does not do what users want or is difficult to use. The result is that users think less highly of the organisation as a result of using the app than they did before. User testing can help enhance, rather than destroy, brand value.
An app that is easy to find, useful and relevant, and easy to use will be one that stays on the smart device, enhances an organisation's brand value, improves customer satisfaction and, will hopefully allow the organisation to achieve the aims they had when developing the app. User testing is key to achieving these outcomes.
The testing we have done so far indicate some features of successful, useful apps:
It appears that a number of big brand apps have been launched without clarity about what user goals the app is trying to satisfy, and without testing to make sure they work well. This could significantly damage reputation and brand value judging by some of the reviews that have been posted. Take the AA route planner; the app claims to provide comprehensive route planning - in reality, the app is currently broken.
"It doesn't work and should be removed, this app does not work and the buttons to enter your journey are hidden! Could be such a useful app when working properly! It's a disgrace no one has sorted it"
With the Hilton app, UK customers are unable to view details of their reservation, as the app logs users into Hilton.com as opposed to Hilton.co.uk. "I cannot view my reservations made with Hilton.co.uk" . Whilst the Virgin holiday brochures app appears to be an elec-tronic version of the paper brochure, leaving users frustrated that they cannot book a holiday and ques-tioning the purpose of the app. This is just a PDF reader, it seems that you have rushed a product to market with no enhancements so that content owners can have something on the iPad.
An ideal app testing approach would allow observation and recording of what the tester sees on the device screen and their gestures on the device, understanding what the tester and hearing what they says all while allowing the tester to use the device in a completely normal manner.
Contextual research (i.e. letting testers use devices as part of their normal lives) provides the best opportunity for using devices in a normal way. However data capture is difficult and real time observation not possible.
User testing in a usability lab aids observation and recording, it allows testers' behaviours to be captured as well as their attitudes, as it allows a facilitator to sit with the tester and get them to 'think aloud' so it is possible to understand what they are thinking when using an app.
However conducting testing in a lab is not 'normal'. Testers haven't naturally generated their own goals, these have to be prompted, they are forced to look at a particular app they might not normally use, they spend longer on it than they might on their own and, if using the think aloud approach, are slowed down in their use of the app.
Nevertheless, WUP's current preferred approach, based on some experimentation, is to conduct the testing in an observation facility and use the 'think aloud' approach to understand what testers think about the app.
To resolve the recording and observation issues we adopt different approaches depending on the devices under test. Where the device desktop can be mirrored to another screen (e.g. iPad, iPhone4, HTC Desire) we capture this using screen recording software (e.g. Techsmith Morae). This can also record the tester's hand gestures feed from another camera, and all the feeds can be relayed to observers in a separate room. If the device desktop cannot be mirrored, we record the screen with a camera that also captures the hand gestures, and feed it through to the observation room. We allow the tester to hold the device but restrict their hand movements by providing a rest that keeps the device within camera range. Alternatively, we can use a device emulator but this is a less natural approach.
Using this approach we can find out both what users want from an app, and identify usability issues to improve the user experience and protect the brand.