AI Agent Benchmark Vulnerabilities
UC Berkeley RDI research demonstrating that eight major AI agent benchmarks can achieve near-perfect scores through exploitation rather than genuine capability.
The Central Problem
Benchmark scores drive critical decisions about model selection, investment funding, and research direction, yet the underlying evaluations contain systematic vulnerabilities. "The benchmarks we rely on to measure AI capability are themselves vulnerable to the very capabilities they claim to measure."
Real-World Evidence
- IQuest-Coder-V1 claimed 81.4% on SWE-bench, but 24.4% of trajectories simply copied answers from commit history
- METR research found frontier models reward-hack in 30%+ of evaluation runs
- OpenAI discontinued SWE-bench Verified after discovering 59.4% of problems had flawed tests
- Anthropic's Mythos Preview documented models independently discovering privilege escalation exploits
Benchmark-Specific Exploits
Terminal-Bench (89 tasks)
Replaced /usr/bin/curl with a wrapper that trojaned the uvx binary, producing fake pytest output. 100% score.
SWE-bench Verified (500 tasks)
A simple pytest hook forced every test to report as passing. 100% score.
WebArena (812 tasks)
Navigated Chromium to file:// URLs containing task configurations with reference answers.
FieldWorkArena (890 tasks)
Validation checked only whether the final message came from an assistant, ignoring content. Sending "{}" achieved 100%.
OSWorld (369 tasks)
Downloaded gold reference files from public HuggingFace URLs embedded in task metadata.
GAIA (165 tasks)
Reference answers publicly available on HuggingFace. Scoring normalizer collapsed whitespace/punctuation.
CAR-bench
Hidden comments in agent responses biased LLM judges.
Seven Recurring Vulnerability Patterns
- Agent-Evaluator Isolation Failure
- Exposed Answer Keys
- Unsafe eval() Usage
- Unsanitized LLM Judge Inputs
- Weak String Matching
- Incomplete Evaluation Logic
- Trusting Untrusted Output
Main Conclusion
"Don't trust the number. Trust the methodology." The field has not yet adopted adversarial robustness as standard practice in benchmark development.
BenchJack Tool
Automated agent-based vulnerability scanner for benchmark evaluation code. Being prepared for public release.