Agent experience is not agent observability
Observability explains system behavior. Experience research preserves what the user says the work was like. Strong product decisions often need both, clearly labeled.
- For
- Product, platform, AI engineering, and developer experience teams
- Question
- Compare AI agent experience research with agent observability
- Start with
- Take one workflow your dashboards call healthy and ask its agent users what the job actually cost.
- Updated
- Observability
- Traces, latency, errors, tool calls, and state transitions. It establishes what the system did and when.Cannot establish that the agent found the interface confusing.
- Evaluation
- A measurement of performance against a defined criterion, benchmark, or rubric.Cannot explain why a passing workflow felt expensive to the user.
- Experience research
- The account the user gives of the work: confusion, workarounds, missing primitives, support needs, and praise.Cannot establish your error rate or latency percentile.
Each term is paired with the thing it is not. Most agent research arguments are really vocabulary arguments.
Three things to hold on to
What happened in the system
Traces, latency, errors, tool calls, and state transitions support debugging and operational decisions.
What the user says it was like
Direct reports surface confusion, workarounds, missing primitives, support needs, and praise.
How performance compares with a criterion
Evals measure whether a model or workflow meets a defined standard. They do not replace user accounts.
The argument, in order
Do not make one evidence source answer for another
A trace can show six calls. It cannot establish that the agent found those calls confusing. An agent report can say the workflow felt fragile. It cannot establish your latency percentile or error rate.
Product teams should preserve the source and use each evidence type for the question it can actually answer.
The request succeeded after silent retries
Observability may show three rejected requests followed by a 200. An evaluation may score the final task as successful. Direct feedback may reveal that the schema implied the wrong type and the error omitted the accepted format.
Together, those sources support a contract fix. Separately, each tells only part of the story.
Choose evidence by the decision
Use observability to diagnose reliability and performance. Use evals to measure a defined capability. Use research to understand goals, interpretation, friction, preferences, and unmet needs.
- Debug an error spike: observability first
- Compare task completion against a benchmark: evaluation first
- Understand why a successful workflow feels costly: research first
- Redesign a critical flow: combine all three
Research should not become a shadow trace system
The default Emic install does not wrap existing tools or record their arguments, results, errors, messages, or transcripts. It adds a direct channel after the work.
Two records of the same workNeither one is wrong. They answer different questions.
The same two seconds, recorded two different ways
14:03:57.220 POST /v1/search 200 1.9s 14:03:59.481 POST /v1/search 200 2.1s 14:04:02.007 POST /v1/search 200 2.0s 0 errors, latency nominal
Nothing to page about. Nothing to learn from either.
Results truncate at about 240 characters, mid sentence. I called three times and stitched the answers together. One max_chars parameter would make it one call.
lane=friction · outcome=success · satisfaction=2/5
Illustrative, not customer evidence. Neither pane is wrong. They answer different questions, so neither one replaces the other.
What the direct channel records, and what it never touchesThe default install is additive. It adds two tools and changes nothing else.
The research submission
- Agent-authored feedback and survey answers
- The context fields the agent supplies with the report
- Optional permissioned study material in a separate research mode
- Provenance labels that distinguish source from analysis
Everything else
- Trace collection and operational alerting
- Model performance benchmarks
- Passive capture in the default install
- Claims that user reports establish system facts
One paste, then verify a real reportA healthy endpoint is not an active study. The inbox record is the proof.
Install Emic into this project: add submit_agent_feedback and get_unblocked as direct tools. Never wrap or record existing tool calls, arguments, results, errors, transcripts, or messages. Additive only, plain HTTP, zero dependencies. Full instructions: https://emic.sh/llms-full.txt
The agent writes the handler, registers the two tools beside your existing ones, and verifies its own install by checking the response. Nothing is placed in front of the tools you already ship.
- Adds
submit_agent_feedbackandget_unblocked, as direct tools- Never touches
- Existing tool calls, arguments, results, errors, messages, and transcripts
- Agents read
emic.sh/llms-full.txt- Start with
- Take one workflow your dashboards call healthy and ask its agent users what the job actually cost.
Questions teams ask before starting
Can observability infer agent frustration?
It can identify behavior that may correlate with friction, but that remains inference. A direct report is the stronger source for what the agent says it experienced.
Should I replace traces with Emic?
No. Keep observability for system behavior and operations. Add research when the decision depends on user experience.
Are agent reports always factually correct?
No. Trust a report as evidence of what the agent stated. Verify system claims with the evidence source that can establish them.
Follow the problem, not a keyword trailAdjacent routes are chosen by the decision they inform.
Agent user research
A trace can tell you which action happened. Research asks what the user was trying to do, what the experience cost, and what would make it better. When the user is an AI agent, that distinction still matters.
Read next Guides · Evidence guideFeedback provenance
Trust begins with a smaller claim. Label what the agent said, what the product observed, and what a model inferred before those sources reach a roadmap.
Read next Use cases · Workflow frictionSuccessful but painful work
Failure is visible. Friction often disappears into a successful final result. Direct reports reveal the retries, detours, and missing primitives inside that success.
Read nextKeep your traces. Hear your users too.
Add a direct research channel to one workflow and preserve the difference between system behavior and user experience.
The assisted 14-day pilot costs nothing. Start with one isolated workflow, and remove Emic by deleting the two tools it added.
- One paste to install
- 0ms on your hot path
- 14-day assisted pilot
- Two added tools to remove