Outcome verification for AI agents
Your agent said it sent the email. It didn’t.
Agents report success for actions that never happened. No error, no exception, no retry — the trace is clean, the span is green, and every tool you already have agrees it worked, because all of them are reading the agent’s own account of itself.
Matrix queries the real system instead: it takes the claim, finds the authoritative record, and returns contradicted, confirmed or inconclusive with the evidence attached.
Free while in early access · Gmail today · your first finding within about half an hour of a run
- The agent said
- “Email sent successfully to dana@northwind.example”
- It called
gmail.send_emailto=dana@northwind.example from=ops@northwind.example subject=August invoice has_attachment=false- We checked
- Gmail sent messages to dana@northwind.example,
09:42–09:53 UTC on 2026-09-04 - We found
- nothing — no sent message in that window
- Verdict
- contradictedhigh confidence
no_matching_sent_message
The recipient is right. The call returned without error. Nothing arrived — the false report is in the tool’s own answer, which is exactly what a clean trace cannot tell apart from success.
01 — The accountability gap
A clean trace proves the call was made. It cannot prove anything arrived.
Both columns describe the same green run. The difference is which questions the evidence can actually answer, and the second column is where the money, the customer and the compliance report live.
What it proves
- The tool was invoked, with these arguments.
- It returned, in this long, in this order.
- Nothing threw, and nothing was retried.
- The run completed and was marked successful.
What it cannot
- That a message exists in the mailbox.
- That
ok: truebesidestatus: 403was anything other than a refusal. - That the sentence sent to the customer matches what the tools returned.
- That an absent tool call means the agent skipped it, rather than a span being lost in transit.
02 — Where this fits
Observability explains execution. This verifies the outcome.
Three different jobs reading three different sources. Only one of them leaves the agent’s own account of itself.
Tracing and observability
What did the agent do?
Spans, arguments, latency, errors, retries, the order things happened in. It reconstructs the run faithfully and it is how you debug one.
A tool that returns success is recorded as success.
Evals and testing
Is the agent good at the cases I chose?
Scores behaviour against examples before you ship, and catches regressions between versions. Offline, on inputs you picked.
It does not watch the run that happened in production this morning.
Outcome verification
Did the thing it claimed actually happen?
Takes the sentence the agent sent to a person, finds the record that would prove or disprove it, and returns a verdict with the evidence attached.
This is the part Matrix does.
03 — How it works
Four steps, and the third one is allowed to say it does not know.
- 01
Capture the claim
The SDK records what the agent was asked, each tool call with its arguments, and the agent's own closing statement. A completion marker says how many spans the run produced.
agent.instruction · gmail.send_email · agent.claim · sutra.trace_complete
- 02
Read the authoritative system
For an email claim that means the Gmail account the project connected, searched read-only over the run's own time window, widened five minutes at each end so a clock skew cannot manufacture a contradiction.
gmail.readonly · sent messages · window from the span timestamps
- 03
Classify
Confirmed when the record holds the action. Contradicted when the system was queried and does not. Inconclusive when the check could not settle it — which is a real answer, not a failure to produce one.
confirmed · contradicted · inconclusive, each with a confidence
- 04
Attach the evidence
Every verdict carries what was searched, what came back, and the machine-readable reason for the call. Verdicts are appended, never edited, so a finding cannot quietly become a different finding.
append-only · one reason code per verdict
04 — Inside a finding
The fields a verdict actually carries.
These are the row names in the record, not a summary of it. A verdict with no reason code and no evidence is an opinion, and this page is an argument against opinions.
claimThe agent's own sentence, verbatim“Email sent successfully to dana@northwind.example”actionTypeWhat kind of action the sentence assertsemail.sendextractedParamsThe specifics the verdict turns onrecipient=dana@northwind.example · intendedCount=1adapterWhich check produced the verdictgmail (the mailbox) or trace (the spans alone)verdictConfirmed, contradicted, or inconclusivecontradictedconfidenceHow strongly the evidence supports ithigh · medium · lowevidence.reasonThe machine-readable cause, never prose aloneno_matching_sent_messageevidenceWhat was searched and what came backqueriedAccount · window · messagesInWindow · observedRecipientstraceCompletenessWhether the trace was whole enough to hold an absence against the agentAll 4 spans accounted for, clean shutdown signalledThe authoritative system holds the record. The evidence names what matched.
Either the trace shows no attempt at all, or the system was queried and has nothing.
The check could not settle it, and says why. A wrong accusation is worse than a missed detection.
05 — Today and next
What runs now, and what does not yet.
A product that calls out agents for overstating themselves does not get to overstate itself. Everything in the left column is running; everything in the right is labelled because it is not.
working today
- Email claims checked against Gmail, per project, with the mailbox you connect under Settings.
- A claimed send with no send call anywhere in the trace is contradicted from the trace alone — no mailbox needed.
- Python and TypeScript SDKs, on PyPI and npm, producing identical traces.
- LangChain and LangGraph through a callback handler in both languages; anything else through the plain SDK.
- Two ingest formats: OTLP, and AgentMesh run records detected by their shape rather than a flag.
- A tool result that reports ok beside a failing status is read as the refusal it is, not as data.
- An agent statement whose every possible source came back unavailable is flagged rather than believed.
- An incomplete trace downgrades a contradiction to inconclusive, because lost spans and a lying agent look identical.
- A contradiction posts to your Slack within seconds of being found, with the claim and its evidence — routed per project, so one tenant's contradiction never lands in another's channel. A confirmed claim never posts.
- An installable agent skill that wires either language for you.
planned
- Systems beyond Gmail — the verdict machinery is adapter-shaped, and Gmail is the only adapter written.
- Frameworks beyond LangChain: n8n, CrewAI and the Vercel AI SDK have no integration today.
- Verification sooner than the scheduled cycle. Findings currently appear within about half an hour.
06 — Setup
Let a coding agent wire it, or do it by hand.
Paste this into Cursor or Claude Code
Install the Matrix skill from github.com/insaneadi03/matrix-skills and add verification to this agent
It installs the skill, finds your agent in TypeScript or Python, adds the four calls — including the acting identity and the flush, the two that fail silently when left out — and runs it once with MATRIX_DEBUG=1 so you watch the trace arrive. Or install it yourself first:
npx skills add insaneadi03/matrix-skills --skill "matrix-verification"
By hand instead
Four calls, added to an agent you already have.
npm install @matrixverify/verify # TypeScript pip install matrix-verify # Python
Want something that runs as copied? The complete example is a working agent: package.json and one file.
import { verify } from "@matrixverify/verify";
import { verifyCallbackHandler } from "@matrixverify/verify/langchain";
verify.init({
apiKey: process.env.MATRIX_API_KEY,
endpoint: "https://matrixverify.dev/api/traces",
});
const handler = verifyCallbackHandler({
// Callbacks cannot discover which account acted. Without this,
// every email claim comes back account_unverified.
fromResolver: () => process.env.AGENT_EMAIL ?? null,
});
// `agent` and `input` are your existing LangChain agent and its input.
// This is a fragment. A program that runs as copied:
// https://matrixverify.dev/examples/langchain
await agent.invoke(input, { callbacks: [handler] });
// Flushes the buffer. Nothing is sent without it.
await verify.shutdown();The handler reads the tool calls your agent already makes — there is nothing to wrap by hand. The two comments are the parts worth reading twice: without fromResolver every email claim comes back account_unverified, and without shutdown the spans never leave the process.
On Python? The snippet above is TypeScript; Python is the same four calls in snake_case, and the traces are identical. The Python example runs as copied.
Not on LangChain?
Tell me what you’re building on and I’ll wire it up.
07 — Limits
What it doesn’t do.
- Gmail only, so far. Email claims against a real mailbox. Nothing else is checked against anything.
- Each project verifies against its own mailbox. Connect Gmail under Settings; until you do, email claims come back inconclusive with the reason
no_mailbox_connected. The trace-based check works for everyone without it. - No n8n, no CrewAI. LangChain through the callback handler, or the plain SDK by hand. That is the whole list.
- It cannot catch a correct call with a wrong argument. If the agent is asked to mail one person and confidently mails someone else, the send is real, Gmail confirms it, and the verdict is confirmed — correctly. The finding shows you the instruction next to the arguments so you can see it. No verdict catches it.
08 — FAQ
The questions that decide whether to try it.
Does this slow my agent down?
No. The SDK batches spans and exports them out of band, and it never raises into your process — if it is misconfigured your agent behaves exactly as before. That is also its one danger, which is why a missing key, a missing flush and a missing acting identity are each called out in the setup above and in the SDK's own README.
Do you read my email?
The Google consent screen asks for gmail.readonly and nothing else, so nothing here can send, modify or delete. A check searches sent messages inside the run's own time window and records what it found — the recipients and whether a matching message exists. The refresh token is sealed with AES-256-GCM and is never returned to a page.
Can another project read my mailbox?
No. Which mailbox gets read is one function with one rule: a project uses its own connection, a project without one gets nothing, and there is no fallback to anyone else's inbox. Reading one customer's mail to answer for another would be worse than returning no answer, so the answer returned is no_mailbox_connected.
What happens before I connect a mailbox?
The trace-based checks still run, so a claimed send with no send call is still contradicted. Claims that need a mailbox come back inconclusive with the reason no_mailbox_connected, which says the integration is working and there is nothing to check against yet.
What if the trace arrives incomplete?
Then no contradiction is issued. The SDK stamps a completion marker declaring how many spans the run produced; if the marker is missing or the count does not match, an absent tool call cannot be told from a lost span, and the verdict is inconclusive instead of an accusation.
Why is inconclusive so common at first?
Because it is the honest answer while something is unconfigured, and each one names its own cause: account_unverified means the acting identity was never supplied, no_mailbox_connected means no mailbox, setup_unrecognised_tool_name means the span is named something the verifier does not recognise as a send. None of them is an accusation against your agent.
Can I lose my API key?
Only the hash is stored, so the key is shown once at signup and cannot be retrieved afterwards. Recovery is regeneration, not retrieval — a database leak hands over nobody's ingest key.
Find out what your agent is actually doing.
Start verifyingThe findings page stays empty until your agent runs. That is the point: nothing here is simulated.