Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem

A 4th Deep Learning for Code Workshop paper describing an LLM agent that performs property-based testing (PBT) autonomously. The agent — implemented purely as a Markdown prompt driving Claude Code — analyzes Python modules, infers function-specific and cross-function properties from code and documentation, writes Hypothesis PBTs, runs them, reflects on failures against a rubric to separate true bugs from false alarms, and emits structured bug reports.

The evaluation is the largest of its kind: 100 popular Python packages (standard library, hand-picked PyPI, and a random sample from the top 5,000), 933 modules, 984 bug reports in 136.6 hours for $5,474 in API cost. Manual review found 56% of reports valid bugs, 32% valid and worth reporting; a 15-point ranking rubric lifted the top 21 reports to 86% valid / 81% worth reporting. Five bugs were reported upstream — NumPy (negative Wald samples from catastrophic cancellation), AWS Lambda Powertools, cloudformation-cli-java-plugin, tokenizers, python-dateutil — with three patches merged.

The agent design is deliberately simple: a six-step loop (analyze → understand → propose properties → write tests → execute and triage → report), a to-do list, broad autonomy, and heavy emphasis on reducing false alarms. The authors name the key limitations honestly: not all 984 reports were human-reviewed, and intent ambiguity means the agent cannot distinguish intentional design decisions from bugs (the requests LookupDict case, where a maintainer had already rejected the same complaint in 2022). Cost per valid bug: ~$9.93.