A Guide To Software Usability Testing

Trying to figure out how to improve a mobile app? Have you observed users actually using it? A great way to evaluate the effectiveness of a mobile app is to observe real people as they complete real tasks in realistic contexts or scenarios. This is called usability testing. With the information collected, app development teams can identify what improvements are needed and quickly implement solutions.

Usability tests can range from being quick-and-dirty, to being highly controlled and scientific. It all depends on the project objectives and the specific research questions being answered by the test. Regardless of the type of test being conducted, researchers should always aim to collect the data as objectively as possible. Doing this requires skill. I've seen data from usability tests be completely invalidated because it wasn't collected objectively--researchers' introduced their own biases and expectations while collecting pertinent information. For example, when collecting preference metrics between two competing designs, some researchers may unconsciously present their personally preferred design in a more favorable manner than the competing design, leading participants' to respond in a way the researcher desires. These type of mishaps can be avoided by being aware of what not to do when conducting a usability test.

After obtaining a representative sample, identifying the core tasks needed to be tested, and setting up the testing environment, researchers should conduct tests in accordance with the research etiquette outline below:

Monitor the Session Impartially

  • Take the attitude that you have no vested interest in the results one way or the other.

  • Present the app or product and each task neutrally.

  • Never indicate through your speech or mannerisms that you approve or disapprove of any actions or comments offered by any particular participant.

    • Be mindful of your speech and body language and never inadvertently provide cues on how to complete a task to a participant.

  • React to mistakes in exactly the same way as you would react to correct behavior.

  • Never make participants feel inadequate by how you respond to their actions.

  • Do encourage participants to freely explore the app or product without concern for “looking good”.

  • If a person has insight about how some other persona group might react to the interface, hear him or her out.

    • For example, if a participant who is an end user believes he or she has insight about who else may benefit from the app or product, take notes on his or her comments.

  • Do not be unduly affected by the performance and comments of the last person observed.

    • Make an effort to “clear the slate” psychologically prior to beginning each testing session.

Don’t “Rescue” Participants when they Struggle

  • Remember: most rudimentarily, researchers should be interested in whether users are able to complete the tasks on their own. This is always true. This can't be discovered if participants are rescued immediately when struggling or expressing confusion.

  • When participants struggle, reassure them that they are doing well and encourage them to verbalize their thoughts and continue as they normally would under real life circumstances.

    • By not letting participants struggle, the opportunity to understand what happens when people get confused and the steps they take to complete the task is lost.

    • It is imperative to identify the aspects of the interface that help people recover from confusion so that those aspects can be made more salient in future iterations. Doing so will keep users from getting confused in the first place.

  • When to provide assistance:

    • If the participant is getting discouraged to the point of shutting down

      • This requires the test monitor to be perceptive of the participant’s attitudes and feelings. If it is sensed that the participant is on the verge of wanting to stop, he or she needs to be rescued before reaching that point in order to not compromise the rest of the test.

    • If it is believed that the participant’s odds of success are improbable

      • If it appears the participant is not likely to discover how to complete the task no matter how much they are enjoying the session, intervene for the sake of not wasting time.

    • When a bug occurs or when a participant’s actions cause malfunctions

    • A phrase to say when rescuing a participant: “Let me give you a hint”

    • Never answer a participant’s question that will help him or her complete the task unless you are rescuing the participant. Instead, encourage the participant to discover the answer on their own as they normally would in a real-life circumstance.

Things to Remember When Conducting the Test

  • It is perfectly acceptable to stop a participant to get a detailed account of his or her actions.

    • For instance, it is reasonable to say something like, “Can you pause for a second? You’ve discovered something helpful and I would like to make a note of it”.

  • Try to record errors discretely

    • Don’t make it obvious that you are recording errors. This will make participants too conscientious of their actions and will discourage exploration.

  • Try to make the test feel like a conversation (this is something that takes lots of practice)

    • Allow the test to be flexible. The test tasks do not have to be completed in the exact order listed in the script.

Hopefully these usability testing guidelines will help you in your next mobile app project.

Have any additional questions?

Let us know by tweeting @chaione.