How to Estimate the Recall of AI Agents with Capture-Recapture

What changed

A technique from fisheries science in the 1890s, capture-recapture estimation, has been tested for measuring recall in AI agents. Applying it to 294 contracts annotated via the CUAD dataset produced recall estimates within 2 percentage points of the true recall, using only a 10% hand audit sample. This approach offers a statistically grounded way to evaluate AI detection performance without exhaustive manual review.

Why builders should care

Recall measurement is critical for understanding what an AI agent misses, but full human audits are costly and time-consuming. Capture-recapture offers a way to estimate recall by comparing overlap between two independent samples of flagged items. For AI operators managing models in sensitive or high-stakes domains, this method lets them confidently assess recall performance with fewer resources, tightening quality assurance without a full-scale review.

The practical takeaway

Operators can leverage capture-recapture to audit AI contract review or compliance models more efficiently. By hand-labeling just 10% of outputs twice independently, they can produce a reliable recall estimate that helps identify if the model is missing critical cases. This cuts down audit workload while providing performance visibility that could drive iterative model improvements or risk management decisions.

What to watch next

As more applications rely on AI for document review and compliance, expect further refinement of capture-recapture and similar statistical auditing techniques. Tracking how broadly this method applies across different domains and model types will be telling. Watch for tools that embed capture-recapture into workflows to automate recall estimation and reporting at scale.

AI Quick Briefs Editorial Desk