Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour
What it does
Keenable AI has open-sourced NEEDLE, a live search benchmark designed to test web search APIs with a query set that refreshes every hour. The key innovation is in its dynamic query list that prevents search agents from gaming the test by preloading answers. Traditional benchmarks risk being compromised because agents can fetch gold labels from public datasets during evaluation, bypassing true retrieval. NEEDLE’s hourly rebuilding query set forces agents to perform actual search, keeping the evaluation honest and relevant in real time.
Why it matters
Search evaluation often breaks when the model or agent knows the answer key. This inflates performance and hides real-world gaps in retrieval quality. NEEDLE disrupts this by continuously updating queries, making it impossible for an agent to merely memorize or download answers. For operators and builders, this means more reliable benchmarking of search APIs and retrieval systems that truly reflect live user scenarios. It raises the bar for testing accuracy, forcing solutions to deliver real search skills rather than shortcut retrieval.
Who it is for
NEEDLE is aimed at developers, researchers, and companies building search agents or AI that rely on external web data. If a project needs trustworthy performance metrics to compare search APIs or tune retrieval components, NEEDLE offers a practical benchmark. This is especially useful for agents with fetch tools or APIs that pull real-time web results, where static tests risk being obsolete or misleading.
The catch
The constant query regeneration demands ongoing infrastructure to update and evaluate every hour, which could complicate integration for some teams. Also, while NEEDLE stops agents from exploiting static gold labels, it does not prevent other forms of prompt or retrieval manipulation. It is a tool to aid fairer benchmarking but not a comprehensive solution for all forms of evaluation gaming.
What to watch next
Watch how NEEDLE influences standards for search API evaluation and whether competitors adopt similar live query benchmarks. Its usage could pressure vendors to improve real-world retrieval instead of optimizing for stale datasets. Also, monitor for integrations with larger LLM evaluation suites or agent frameworks, which can amplify its impact on AI search and agent design.
AI Quick Briefs Editorial Desk