claims-audit-agent — three substrates, one eval

What the numbers say

Frozen 25-claim holdout. Every LLM arm runs the cheap open model qwen/qwen3-32b through OpenRouter, so the comparison is apples-to-apples.

0.000

Fabrication rate — every arm. No finding ever cites a rule or line that doesn't exist.

1.000

Recall on the Vercel AI SDK arm (both modes) — every injected defect caught.

0.71 → 0.94

AI SDK precision, thinking off → on. Reasoning cut false positives 6 → 1 at full recall.

$0.0006

Cost per claim for the AI SDK arm — the whole 25-claim holdout for pennies.

Results — frozen holdout (25 claims)

The deterministic engine is the CI-gated arm (measured on every push). The LLM arms are reported, not gated.

SDK / Arm	Thinking	Model	Precision	Recall	F1	Citation	Latency	$/claim
engine (deterministic)	n/a	n/a	1.000	1.000	1.000	1.000	<1 ms	$0.00000
langgraph · openrouter	off	qwen3-32b	0.750	1.000	0.857	1.000	3.6 s	$0.00018
langgraph · openrouter	on	qwen3-32b	1.000	0.600	0.750	1.000	1.5 s	$0.00016
claude-agent-sdk · openrouter	off	qwen3-32b	1.000	1.000	1.000	1.000	54.3 s	$0.00324
vercel-ai-sdk · openrouter new	off	qwen3-32b	0.714	1.000	0.833	1.000	15.3 s	$0.00063
vercel-ai-sdk · openrouter new	on	qwen3-32b	0.938	1.000	0.968	1.000	11.9 s	$0.00059

These are measured numbers on the committed holdout, scored by the repo's canonical Python harness (unchanged). The reasoning delta is orchestration-dependent: on the autonomous AI SDK loop, thinking-on cut over-flagging (precision 0.71 → 0.94) at full recall — the opposite of the LangGraph arm, where thinking-on traded recall for precision.

Three substrates, honestly compared

Same tools, same prompt, same holdout. Here's where each one helps and where each one fights you.

claude-agent-sdk

Autonomous in-process tool loop

schema: Python type hints → tool schema; Pydantic Finding
loop: Model-driven; least code, most capability
observe: Opaque — control flow lives in the model
fights: Hard to bound / unit-test offline; a CLI subprocess per claim (~54 s, flaky vs 3rd-party endpoints)

LangGraph

Explicit, fixed graph

schema: JSON-schema forced tool; Pydantic core
loop: Data you can read; an evidence_check node makes fabrication structurally impossible
observe: Best — every node is inspectable; runs offline with an injected model
fights: Most wiring; you hand-build the control flow

Vercel AI SDK

Autonomous loop, TypeScript

schema: Zod Finding, field-for-field with Pydantic; .strict() = extra="forbid"
loop: generateText + stopWhen: stepCountIs — the bound is one explicit line
observe: Typed steps[] / usage per call; a first-class offline mock model
fights: No evidence_check seam (like the SDK arm, it trusts the model + strict schema); an unknown model id trips a maxOutputTokens compat warning

One tool is also wired through the AI SDK's MCP client to a separate sparql-mcp server (SSE) — proving an external MCP tool drops into this arm through one standard interface, without touching the scored audit loop.