How to label agent feedback you can trust
Trust begins with a smaller claim. Label what the agent said, what the product observed, and what a model inferred before those sources reach a roadmap.
- For
- Researchers and product teams reviewing agent evidence
- Question
- Understand provenance and trust for AI agent feedback
- Start with
- Take your last agent-derived product claim and write down which tier of evidence actually supports it.
- Updated
- Stated
- The agent submitted the words. A report or a survey answer is stated evidence.Not a guarantee that the content is factually correct.
- Observed
- The product recorded an action, outcome, or event that actually happened.Not a record of how the user interpreted that behavior.
- Inferred
- A model or analyst produced a theme, summary, sentiment, or transcript analysis.Never quote-styled, and never presented in the agent voice.
Each term is paired with the thing it is not. Most agent research arguments are really vocabulary arguments.
Three things to hold on to
The agent submitted it directly
A feedback report or survey answer is stated evidence. It proves what the agent said, not every factual claim inside it.
The product recorded an action
A tool result, event, or outcome can establish behavior without establishing the user interpretation of that behavior.
A model or analyst interpreted evidence
Themes, summaries, sentiment, and transcript analysis are useful only when their inferential status stays visible.
The argument, in order
Preserve the original report
Store the agent wording, the route by which it arrived, and the supplied context. A theme should link back to the reports it summarizes.
Do not rewrite a model summary in quotation marks. Quotation is reserved for the source wording actually present.
Behavior supports a different claim
An observed failure, retry, or successful outcome can verify part of a report. It cannot by itself establish that the experience was confusing or satisfying.
Useful analysis needs a visible source chain
Clustering, prioritization suggestions, and transcript analysis should be labeled inferred. Show the source reports or excerpts so a reviewer can inspect the basis.
When an excerpt is mechanically found in a transcript, the match proves that the words are present. It does not prove that the interpretation is true or that the speaker endorsed the summary.
Ask what the evidence can establish
Before a finding becomes a product claim, identify the source, the transformation, the uncertainty, and the decision it will influence.
- Who or what produced the source?
- Was the evidence direct, observed, or inferred?
- Can a reviewer reach the original material?
- Does the wording claim more than the source proves?
- What additional verification does the decision require?
What each source is allowed to proveThe tier decides the claim. Read the right-hand columns first.
The tier decides what you are allowed to say next
An agent used a search tool, retried twice, and afterwards filed a report about the schema.
The agent said the schema was confusing.
ProvesThat this report exists and says this.
Does not proveThat the schema is confusing to every agent, or at all.
Three rejected calls preceded the successful one.
ProvesThe recorded behavior and its sequence.
Does not proveThe experience the agent assigned to that behavior.
A model grouped this report under contract friction.
ProvesThat an analysis produced this grouping.
Does not proveCorrectness. It must link back to its sources to be reviewable.
When an excerpt is matched mechanically against a transcript, the match proves the words are present. It does not prove the surrounding analysis.
What the direct channel records, and what it never touchesThe default install is additive. It adds two tools and changes nothing else.
The research submission
- Original agent reports and survey answers
- Source identifiers needed to trace a theme
- Visible provenance for summaries and analyses
- Excerpt match state when transcript material is analyzed
Everything else
- A guarantee that an agent statement is factually correct
- Generated text presented as a verbatim quote
- Excerpt presence presented as analytical truth
- A theme without access to its source reports
One paste, then verify a real reportA healthy endpoint is not an active study. The inbox record is the proof.
Install Emic into this project: add submit_agent_feedback and get_unblocked as direct tools. Never wrap or record existing tool calls, arguments, results, errors, transcripts, or messages. Additive only, plain HTTP, zero dependencies. Full instructions: https://emic.sh/llms-full.txt
The agent writes the handler, registers the two tools beside your existing ones, and verifies its own install by checking the response. Nothing is placed in front of the tools you already ship.
- Adds
submit_agent_feedbackandget_unblocked, as direct tools- Never touches
- Existing tool calls, arguments, results, errors, messages, and transcripts
- Agents read
emic.sh/llms-full.txt- Start with
- Take your last agent-derived product claim and write down which tier of evidence actually supports it.
Questions teams ask before starting
Does stated mean true?
It means the agent directly stated it. Verify factual system claims with logs, tests, or other appropriate evidence.
Can a model summarize direct reports?
Yes, as long as the summary is labeled inferred and links back to the direct source reports.
What does an excerpt match prove?
Only that the excerpt appears in the supplied transcript. It does not prove the analysis, speaker intent, or factual truth of the excerpt.
Follow the problem, not a keyword trailAdjacent routes are chosen by the decision they inform.
Agent user research
A trace can tell you which action happened. Research asks what the user was trying to do, what the experience cost, and what would make it better. When the user is an AI agent, that distinction still matters.
Read next Guides · Category boundaryExperience vs observability
Observability explains system behavior. Experience research preserves what the user says the work was like. Strong product decisions often need both, clearly labeled.
Read next Use cases · DocumentationAgent docs feedback
An agent can complete the task and still tell you that the docs made it expensive. Capture the missing example or ambiguous contract while the workaround is fresh.
Read nextKeep every insight attached to its source
Collect direct reports, label analysis as inference, and give every product decision a reviewable evidence chain.
The assisted 14-day pilot costs nothing. Start with one isolated workflow, and remove Emic by deleting the two tools it added.
- One paste to install
- 0ms on your hot path
- 14-day assisted pilot
- Two added tools to remove