Woman observing an older man using a tablet

How AI Helped with Our Latest Usability Test

Don’t look at us like that. It’s not what you think. 

AI has been a huge time-saver and immensely helpful in our latest usability test. The last case study we conducted is for software that hasn’t been released yet, so apologies for omitting the name and being a bit vague about the actual project. 

Our last usability test was for a multi-brand B2B marketplace where professional buyers (distributors, commercial contractors, and the internal sales team) can place orders for their own inventory or on behalf of their customers. 

Don’t worry. We used real human participants and a real human moderator. We’re not using AI agents in place of human participants. 

That’s not the hard part anyway. The challenging part of moderated usability testing isn’t the test itself. It’s the analysis and the synthesis – breaking things down into actionable insights that improve your software. 

So this time, we ran the entire analysis phase with an AI assistant as a working partner, and we kept an honest ledger about what it got right and what we had to correct. It made the process feel easier and more accurate. The things it got wrong were the kinds of things that require human judgement.

The Setup

We fed Claude the raw transcripts, moderation script, and the rubric. (Technically, we used an AI recorder for the transcripts, too.) We set this ground rule up front:

This is generally what our rubric looks like. For each participant, we have a page in OneNote with a table that looks like this.

QuestionExpected ResponseActual Response
How would you buy this product?I would pick the color I like and then click “add to cart.”I would click “add to cart.”
Let’s say you want to buy a red shirt. Go ahead and do that.Click red swatch > Click “Add to Cart” button“Where’s the red? Oh, there it is.” 
Clicks red swatch > Clicks add to cart

We do it this way, even without AI. It highlights where actual responses or behaviors did not match what we expected. Whether or not the participant completed the task or was able to answer the question is still subjective. Here, we’re just recording actual responses. 

For our AI assistant, it sets the table and lets it know what responses we’re looking for. 

Observation only. No judgement. “Participant 1 said this.” “Participants 1, 3, and 4 struggled to complete task 5”. No judging how users felt about the interface. That’s not relevant to this study. No recommendations. 

I also fed it sample reports we’ve done in the past, and it already had our branding information. 

What it got right

Fast session summary

After every session, Claude provided a quick, structured summary. It included what participants did, where they struggled, and what they saw that didn’t align with their expectations. 

Cross-session pattern-finding

This one was super helpful. In the past, we would have to go back through our notes to find patterns between multiple sessions. With Claude, I was able to just query things like, “Which participants struggled with task 5. Give me timestamps, and what they said or did to indicate they struggled.” And it was just faster to do it that way than to go back through our own notes. 

Structured Deliverables

Once we chatted with it, providing corrections and additional things we found, it was easy to have it fill our report templates with data from the studies:

  1. Usability test findings report
  2. Prioritized Backlog

What we had to correct

AI helps us synthesize faster, but it lacks human judgement. It’s getting better, and maybe it won’t always be this way, but for now, a human still needs to review findings.. and we are ok with that!

It Didn’t Account for Limitations in the Prototype

Our test included faked instant-search results. Users said they would search for something else, and would expect to see matches related to that search. Claude flagged it as a task failure. It wasn’t really a failure in the UX – just a prototype limitation.

It presented statistics that were not statistically significant

We only tested with 5 users. That’s not enough for quantitative statistics, like task success rate, error rate, or time on task. You need more participants to report on those kinds of metrics meaningfully. This was a qualitative study. 

It needed us as the domain and priority guardrail

It made a few mistakes. It occasionally attributed a quote or struggle to the wrong participant, and we flagged it after reviewing transcripts. It also didn’t have anything concrete to go on in terms of priority. So in the backlog, we set the priority manually.  

What made it work

If you want to try this in your own usability test, here’s what we would recommend:

  • Don’t use AI participants (unless AI is one of your personas, of course)
  • Feed it the data. All the data. The transcripts for the usability tests, stakeholder sessions, all of the design sessions. Everything. We would have done that with this test, but the client owned the stakeholder and design session recordings, and we couldn’t access them. It would have helped with prioritizing. 
  • Keep the human as the analyst and the AI as the synthesizer. Humans should push back, and ask follow-up questions
  • Force attribution. Map every finding back to actual participants and timestamps. 
  • Let it handle formatting the reports. 

What we'd do differently next time

A couple of lessons learned:

  • Feed it all of your research data, not just the usability test transcripts. Feed it the design sessions, the status meetings, the stakeholder feedback. We think this could help make AI a better judge of priority.
  • Separate “prototype defect” from “usability finding” as feedback for the individual session read-outs. This way, you don’t have to deal with them later
  • Keep a correction log of every time you overruled the AI assistant. This makes the end-deliverables more accurate.

All-in-all, we’d absolutely use an AI assistant to help synthesize usability test data again. And we intend to continuously refine our process as we do more of them.