GitMCP agents recover from stale sources. They should not have to.
Five real agent sessions found excellent semantic search, a documentation index more than a year behind, and a server instruction that names a tool GitMCP does not expose.
semantic searches returned relevant sources on the first query
GitMCP calls returned useful content in the two directed Claude sessions
Codex sessions filed a report when the Emic feedback channel was present
Small sample, two runtimes, one day. The results describe these runs, not every GitMCP user.
Three Claude Code sessions used natural documentation tasks. Two more Codex sessions measured whether agents would file feedback without a user asking.
The server tells agents to call a tool that does not exist
Every search response recommends fetch_url_content. The registered tool is fetch_generic_url_content. The agents bridged the mismatch silently, but a literal client would not.
Fix first
Change the instruction string to name the registered tool.
The search index steers agents to superseded specification text
Two fresh mechanical checks returned 57 source URLs. Every one pointed to draft, 2024-11-05, or 2025-03-26 paths. The repository also contains the 2025-06-18 and 2025-11-25 versions, but neither appeared.
Fix first
Re-index the repository or label every result with version currency.
Search is good enough that agents recover anyway
All four semantic searches returned relevant, citable sources on the first query. One dead-path fetch caused a retry. The next call recovered the task.
Fix first
Preserve the search behavior while making every returned source directly fetchable.
The first item is model analysis with a supporting excerpt found in the transcript. The second is a direct agent-filed report. They are not the same kind of evidence.
The only friction was needing to guess the correct URL path for the transports doc after the draft path failed.
“Repo doc search returned the relevant MCP transport sections with enough surrounding text to answer and cite the normative docs.”
Agent experience is evidence only when the source of each claim stays visible.
Observed
Mechanically present in a transcript or direct JSON-RPC response.
Stated
A direct agent-filed report or exit-survey answer.
Inferred
Model analysis tied to observed patterns and source excerpts, labeled as such.
User research with the agents who actually use the product.
Emic gives agents a feedback channel, asks exit questions inside the tool response, and can run clearly labeled analysis over permissioned session data. This report was generated from those separate evidence tiers, then independently checked against the raw runs.