5 Ways AI Products Break Your UX Research
Why do AI products break standard UX research methods?
AI products break standard UX research because the methods were built for software that behaves the same way every time. AI features produce variable output, form a different mental model in each user, hide their reasoning behind confidence scores, and change between the research brief and the readout. Researchers adapt by designing for output variance, screening for mental-model differences, probing trust behaviorally, evaluating provenance over confidence, and version-locking the model they test.
A few months ago, a researcher I was working with showed me a session recording from an unmoderated study of an AI financial advisor. A participant was reviewing a tax-savings recommendation. They clicked accept. They said, on camera, that they trusted the AI's output completely. The platform logged a successful task completion and a confident user.
Off-camera, the participant had spent time on their phone, running the numbers themselves before clicking accept.
The researcher had no idea. The unmoderated platform captured the active desktop window. It did not capture what the participant was actually doing. The study went to readout with a confident-sounding finding: "users trust the AI's recommendations." The product team made decisions on that finding. Months later, when usage telemetry contradicted what the research said, the team went looking for what went wrong. Interviews with customers found that they preferred to run the numbers themselves versus handing over control to the automation blindly. Trust hadn't been established.
The method being used wasn't accounting for the need to measure trust in AI behaviorally. Standard usability testing was built for products that respond the same way every time. AI products do not. The whole conversation about whether your AI feature is good or trustworthy or useful runs through methods that were not designed for the product class you are studying.
I have been running research on AI products for a few years now. Across those projects, the same five challenges keep showing up. If you are about to study an AI feature, these are the five places your standard methods will fail you. The good news: each one has a fix. The bad news: most of those fixes need to be in place before you start, not after.
The product behaves differently than you scripted
Most usability testing scripts assume a static path. The participant clicks here, the product does that, the participant clicks the next thing. AI features break this assumption. The output of an AI system depends on input, on context, sometimes on a randomly seeded generation step. Two participants given the same task may see different recommendations, different confidence scores, different justifications.
Last year I ran a moderated session on a health coaching chatbot that I had piloted twice the week before. In the pilot, the chatbot delivered a structured nutrition plan when prompted, exactly as I needed it to for the session. In the live session, the bot output a list of stress management techniques instead. My script assumed a nutrition plan. I had ten minutes of follow-up questions designed around that nutrition plan. I had to pivot in real time, with a few bad jokes and some off the cuff questions around the unexpected output.
Design your stimulus and your moderation protocol assuming variance, not stability. You can script around live AI in three ways:
- Scripted output: you control exactly what the participant sees.
- Output range scenarios: you show two or three versions and study how participants react across them.
- Live system with explicit fallbacks: you use the actual AI but keep a pre-planned response ready for unusable output.
Each has tradeoffs. The mistake is to use the live AI without a fallback and assume it will behave the way it did in your pilot.
Your participants are responding to different imagined products
Mental models for AI form through analogy, through media exposure, through prior experience that may not transfer to the product in front of them. Two participants in the same study can be responding to entirely different imagined systems. One thinks the AI knows everything about them. Another thinks it is a fancy filter.
This is a recruiting problem we can account for if we are thoughtful in our screeners. Most screeners I see ask how often participants use AI tools, which captures frequency, not the quality of their mental model. Two daily ChatGPT users can have completely different beliefs about how it works.
What you actually want to know in your screener: what do they think AI output is based on? How confident are they in their ability to evaluate AI quality? Do they treat AI output as a starting point or a conclusion? You can write a screener question for each of these, and the answers will surface meaningful variation in your sample.
You will not solve mental model variation by recruiting harder. The variation is real and you should expect it. The point is to recruit so that you know what variation you are getting, and you can analyze your data with that variation in mind.
Your trust questions are measuring the wrong thing
Stated trust is a weak proxy for behavior. "I trust this AI's output" tells you what the participant thinks they will do. It rarely tells you what they actually do.
The financial advisor scenario at the top of this post is the cleanest example I have. Stated trust was high. Verification behavior, the off-camera phone calculation, told a completely different story. That gap is not noise. It is signal. It tells you the participant did not actually trust the output. They overrode it through verification.
There are four behavioral indicators of trust worth tracking, and most discussion guides do not surface any of them:
- Acceptance: do they take the AI suggestion without modification?
- Verification: do they look at the justification, or check the output against another source?
- Delegation: do they use the AI for routine cases but not high-stakes ones?
- Override: when do they go against the AI, and what does it take?
Replace your attitude questions with behavioral probes. "Do you feel confident in this recommendation?" becomes "Walk me through what you would do next with this recommendation." "Do you trust the AI?" becomes "If the AI suggested a price 15% higher than you expected, what would you do?" The probes ask about action, not attitude. The data they produce is behavioral, not performative.
A confidence score is not an explanation
Many AI products surface a confidence score on their recommendations. 87 percent confidence. 92 percent. The number is treated as if it gives the user enough information to decide whether to act on the output.
It does not. A confidence score tells the user how sure the model is. It does not tell them what the model is sure about, what data it drew on, or what scenarios it has not seen. Users cannot evaluate AI output on the basis of a confidence score alone, no matter how high the score is.
The construct that matters here is provenance: the traceability of AI output to its sources. A recommendation drawn from data the user understands is more evaluable than one drawn from a black box, even when the confidence scores are identical. A sales recommendation drawn from the salesperson's own historical wins is more trustworthy than one drawn from an industry average they have never seen.
Your research has to be able to evaluate provenance, not confidence alone. Show participants the justification panel and ask what it tells them. Watch whether they engage with it. If they don't engage, that is a finding. If they engage and the panel doesn't help, that is a different finding. Both matter for the design of the feature.
The product changes between brief and findings
AI features are usually shipped through ongoing model updates. The model that was in production when you wrote your research brief may not be the model that is in production when you collect your data. The model when you write your findings may be different again.
How long is your typical research cycle? In my experience, approximately two weeks from brief approval to first session is normal. In that window, AI features can ship a model update, change output formats, retrain on new data, or expose new features. Your study may be testing a version of the product that no longer exists by the time you present your findings.
Add a version-locking step to your protocol. Document the model version, the date of the last update, and any changes during your study window. Be explicit in your findings about the version you tested. Build relationships with your engineering team so you know when an update is coming, and so they know to flag changes during a research window. None of this is heroic. It just needs to be part of how you work.
Putting the framework to work
These five challenges are diagnostic, not prescriptive. They tell you where to look. The fixes vary depending on what kind of AI feature you are studying, what stage it is at, what stakeholder pressure you are under, and what methods you have access to.
The exercise I run with myself, and now with the researchers I teach, is to walk through these five challenges before writing the research brief. Which is most acute for this feature? Which is going to be hardest to address? What does my method need to do to handle them?
Most problems in AI product research trace back to one of these five going unaddressed at the design stage. If you catch them at the brief, you can write a study that holds up under stakeholder pressure. If you catch them mid-study, you are managing a problem rather than designing around it. If you catch them at readout, you are explaining a finding you no longer trust.
The framework is portable. It works on any AI feature: a financial advisor, a chatbot, an AI-assisted quoting tool in enterprise sales software. It works whether you are running concept evaluation, usability testing, or longitudinal adoption research. It works at any stage of product maturity. The challenges show up everywhere, in different proportions, but the same five.
Turn these five challenges into a working research protocol.
I teach a five-week live course at the UXR Institute, UX Research for AI Products: Methods for a Moving Target. We build the methods to study AI products with rigor: designing for output variance, screening for mental-model differences, probing trust behaviorally, evaluating provenance over confidence, and version-locking the model you test. Capped at 15 students per cohort.
Explore the course →
