“Usability Testing” Continues To Be Confused With “User Feedback” and Other “Testing”

In 2013, I wrote a blog post called “Focus Groups Vs Usability Testing“. Eight years later, the confusion only seems to have gotten worse. Not only do people confuse Focus Groups with Usability Testing but many also confuse User Feedback and even QA Testing with Usability Testing.

In the early stages of a client engagement, we often ask if any usability tests have been performed on the software product we’re being asked to redesign. Sometimes we get responses such as “Sure, our tech support team gets feedback from our customers all the time.” or “Yes, our marketing agency tested the product with focus groups.” or “Our team of testers test every build before each product release.” Unfortunately, none of these are usability tests.

How User Feedback is not Usability Testing

Getting user feedback is great but it often comes in the form of a compliment or a complaint. While this is certainly useful, there may be many other opportunities for improvements that usability testing can reveal. In addition, we’ve also seen many instances where the users may be internal employees who are cautious about complaining about a product because they either don’t want to seem like a complainer or they’re afraid to insult someone in a higher position. This is why we encourage our clients to allow us to perform both user shadowing and usability tests without anyone else involved. Sure, we’ll record sessions so the client can playback a video of exactly what we observed but, when a user knows they’re being observed by an audience, they simply behave differently.

The other issue with getting user feedback is that it is often gathered by asking for the feedback, usually in the form of a survey or a user feedback form. Unfortunately, when you ask someone “What do you think of this?” you may as well be asking them to “Find something wrong with this.” In other words, the user feels they are being tested — not the product design — and they think have to find something wrong otherwise they’ll fail the test. In usability testing, we stress how we are testing the product’s design – not the users – so there is nothing they can do “wrong”. This simple statement that is said to the user prior to testing, helps the user relax, focus on the tasks they’re being asked to perform, and behave much more naturally.

How QA and UAT Testing is not Usability Testing

There have also been times when we’ve asked about usability testing and we’re told how the product has been thoroughly tested and is fully functional and bug-free. In other words, they hear the word “usability” and they think “functionality”. However, a product can be fully functional while also being nearly impossible to use. In Don Norman’s book, The Design of Everyday Things, he discusses how there’s an epidemic of poor door designs, which has now been coined as “Norman Doors” (see video below). In one example, the doors are fully functional but because of the poor design, numerous people were getting stuck between two sets of doorways. The point is, you can have a fully “functional” product that is nearly “unusable”.

How to overcome the confusion

The solution to the confusion we continue to see regarding usability testing is for those of us in the User Experience profession to do a better job at explaining and educating people on what it is, what it is not, and why it is so valuable. Much the way we UXers try not to blame the users for not understanding how to use a product, we shouldn’t blame non-UXers for not understanding what usability testing is. We need to do our best to get on our soapboxes and teach the masses. In fact, you do your part by sharing this post!


Why The Google Places API Search Failed Our Usability Tests

Whenever we perform usability tests, we inevitably discover something we didn’t expect — which is exactly why we test. In this case, we were designing a quoting system for insurance agents and the primary success criteria for the project was to “increase quote submissions by making the quoting process dramatically faster and easier for the agents”. When it comes to streamlining any complex process, it’s usually a lot of little tweaks that add up to significant time savings. For this quoting system, one of those things was the “Business Name” field. During our user shadowing sessions with the users, we observed how laborious and error-prone it was for users to enter the business name every time they created a quote. So, we suggested implementing the Google Places API Search, on the Google Maps Platform, which we’ve used on many other applications before with great success.

For this application, our stated hypothesis for using Places API was this:

“We believe that by replacing the text input field with the Google Places API search feature, we will optimize the overall efficiency of this required field for the following reasons:

  1. Redundant data entry will be reduced because the user can select a business from the results set after typing just a few characters
  2. User input errors will be reduced because the users will select a business from the search results
  3. Data quality and data mining capabilities will improve because businesses are motivated to keep their Google listings up-to-date because so many customers use Google
  4. Google’s Instant Autocomplete functionality is familiar to most users because of Google’s widespread usage and adoption”

Google’s ‘Familiar’ Instant Autocomplete Results

Since its official launch in 2008, most people have grown very familiar with how Google’s Instant Autocomplete search results work. You start typing something in and Google immediately starts to show you matching results that you can select. With Google’s Place’s API (shown to the right), a user begins to type a business name and Google immediately starts to display matching business names that are closest to the user’s location and they have the option of either selecting from the Instant Autocomplete results or selecting the option to create a new business listing.

The Surprise

In our initial usability tests, not a single agent selected a business from Google’s Instant Autocomplete results that displayed right in front of them as they typed. Instead, every one of them either typed the full business name or they copied and pasted the business name into the field from another data source. To be clear, we are not saying that Google’s Instant Autocomplete results are poorly designed, we are saying that when we implemented the feature in the redesign of an insurance quoting system, this ‘familiar’ feature was not used by agents as we expected it to be used. In fact, we were so baffled by the failed test results that we decided to test it again with non-agents just to see if we could narrow down the root cause of the failed test. As we suspected (and hoped), all non-agents selected from the search results rather than type the full business name or copy and paste it into the field. 

The Lessons Learned

So why did such as familiar feature fail our usability test? We believe the primary reason was “Learned Behavior”. The agents we tested all use many different insurance quoting systems and not one of them uses Instant Autocomplete results for the Business Name field. Instead, as flawed as it may seem, they all use a simple free-form text entry field that requires the users to type in the full business name. So, the users are simply used to typing the full business name instead of selecting it because within the context of an insurance quoting system, that’s what they’ve always done.

The Solution

So how do we change the way the agents see this Business Name field so that they use it more as a search box instead of a text entry field? We took a page from the CueActionRewards (CAR) Model in Behavioral Design, which is a proven design framework for inducing user habits.

How the Cue-Action-Reward (CAR) Model has been used to induce habits.

Source: The book “Human Behavior is Programmable”

In the case of this one Business Name field, we needed to make it visually obvious (Cues) that the results were ‘selectable’ so that it encouraged users to click (Action) a result because it would mean less typing, less errors, and it would provide an opportunity to pre-fill a lot of data within the quoting system based on the the unique ID of the selected business (Reward!).

Here’s what we did:

    • Added the word “Search” to the field label name and added a search icon as Cues to get agents to use this field as a search box rather than a text entry field.
    • Added the “Select a Matching Business” (Cue) to encourage a selection (Action) from the results set.
    • Added radio buttons (Cue) to the results to further encourage a selection (Action).
    • Colored the business names in the results set blue (Cue) to help indicate that they are clickable.
    • The Reward is then fulfilled by typing less and on the next screen a message informs the user that data was automatically pre-filled for the selected business, which we hope will encourage a repeat of the same use then next time the agents uses the Business Name field.

All This For a Single Form Field?

Yes. In order to “increase quote submissions by making the quoting process dramatically faster and easier for the agents” we needed to get this first form field right.

But First, Prototype – Correcting Mistakes Before They’re Made

Source: quotesondesign.com

Sometimes ideas sound great in theory, but don’t work well in practice. That’s why prototyping with actual users is essential. We usually think of prototyping in terms of software, but it has real world applications, too.

Read more

Sometimes, To See “The Big Picture”, You Need To Draw A Big Picture

To truly understand and design the best possible information architecture for an application or website, find the biggest whiteboard in your office and start drawing.

Read more

The Disappearing Email Preheader

For the last few years, marketers have used ‘teaser text’ to give users a sneak peak of an email’s content. Email clients on mobile as well as desktop devices pull that teaser text from the first line of the email body. Lately, we’ve found that the teaser text is often invisible in the email. Knowing that 56% of emails are opened on mobile devices1, we can guess why.

1Source: https://litmus.com/blog/mobile-friendly-email-september-2016-email-market-share

Read more

E-commerce Shopping Cart Usability Research Findings

The Nielsen Norman Group just release the results of a Shopping Cart Usability study called “Decision Making in the E-Commerce Shopping Cart: 4 Tips for Supporting Users“. Read more

How UX Design Plays a Vital Role in Preventing Medical Errors

In a recent article published in Fast Company’s Co. Design, Jonathan West of the Helen Hamlyn Centre for Design was interviewed about a project he worked on called DOME (short for “Designing Out Medical Error”). The entire DOME team is comprised of designers, clinicians, psychologists, and human factors experts all focused on studying and designing solutions that could reduce medical mistakes made in hospitals. What is especially interesting is how the article sights many of the same methods and activities UX Team often uses on our software design projects Read more

Every Dollar Invested In Ease Of Use Returns $10 To $100

For most of my career, I have been standing on my soapbox preaching the importance of User Experience to anyone that would listen. Usually it’s during an initial sales presentation where I’m trying to convince a client why they should spend more time and focus on the upfront design work of their software because it will save them a ton of time and money on the development work. Read more

The UX Process and UX Design Principles

A web application’s design and usability is just as important as that application’s functionality. If our clients, their users, their partners, and their vendors can’t immediately figure out how to use a web application, then we haven’t done our job correctly. Read more

The Blur Test: Go Daddy

The Blur Test is an old art school technique used to reveal a design’s focal point and visual hierarchy. Let’s see how Go Daddy holds up to The Blur Test. Read more