Raw capture — raw/2510-09907v1.md

url: https://arxiv.org/html/2510.09907v1 date_fetched: 2026-10-18

The 4th Deep Learning for Code Workshop

Agentic Property-Based Testing:

Finding Bugs Across the Python Ecosystem

Abstract

Property-based testing (PBT) is a lightweight formal method, typically implemented as a randomized testing framework. Users specify the input domain for their test using combinators supplied by the PBT framework, and the expected properties or invariants as a unit-test function. The framework then searches for a counterexample, e.g. by generating inputs and calling the test function. In this work, we demonstrate an LLM-based agent which analyzes Python modules, infers function-specific and cross-function properties from code and documentation, synthesizes and executes PBTs, reflects on outputs of these tests to confirm true bugs, and finally outputs actionable bug reports for the developer. We perform an extensive evaluation of our agent across 100 popular Python packages. Of the bug reports generated by the agent, we found after manual review that 56% were valid bugs and 32% were valid bugs that we would report to maintainers. We then developed a ranking rubric to surface high-priority valid bugs to developers, and found that of the 21 top-scoring bugs, 86% were valid and 81% we would report. The bugs span diverse failure modes from serialization failures to numerical precision errors to flawed cache implementations. We reported 5 bugs, 4 with patches, including to NumPy and cloud computing SDKs, with 3 patches merged successfully. Our results suggest that LLMs with PBT provides a rigorous and scalable method for autonomously testing software. Our code and artifacts are available at: https://github.com/mmaaz-git/agentic-pbt.

1 Introduction

Property-based testing (PBT) is a software testing paradigm that aims to verify whether a general property holds over a predefined input domain. In contrast to traditional example-based testing, it does not require specific examples of expected outputs. Instead, PBT allows the developer to define invariants that should hold for all inputs, which are then checked on a diverse set of automatically-generated inputs. A property might require that the output of a function is always non-negative, that some and commute for all (), or that an interpreted program has equivalent semantics to its compiled form. Popularized by QuickCheck in Haskell (Claessen and Hughes, 2000), there now exist PBT libraries for many of the common programming languages, including Python’s Hypothesis (MacIver et al., 2019), the focus in this work. PBT can be more robust than example-based testing which inherently relies on the developer to anticipate edge cases (MacIver, 2019).

Despite its theoretical appeal, property-based testing is less popular than example-based testing due to the challenge of identifying meaningful properties to test (Goldstein et al., 2024). Specifying meaningful properties, especially as codebases get more complex, often requires significant domain expertise and time investment. Recent advances in large language models (LLMs) have demonstrated remarkable code understanding, which makes it possible to automatically mine for properties. However, existing work on using LLMs for PBTs focuses on a single function at a time, and has only a single generation step.

In our work, we introduce agentic property-based testing, an approach leveraging the multi-step reasoning capabilities of coding agents and the rigor of PBT. We develop an agent that can autonomously crawl through and understand entire codebases, look for high-value properties (including across functions and classes), write PBTs and run them, and then analyze failing tests for validity. After the agent runs for several rounds, it either surfaces a bug, or states it is unable to identify one.

We demonstrate the effectiveness of agentic PBT through a large-scale evaluation across the Python ecosystem, testing 100 popular Python packages with billions of downloads. Manual evaluation of the bug reports generated by our agent showed that it successfully identified genuine bugs in several widely-used libraries: for example, our agent autonomously finds a bug in NumPy, the pre-eminent numerical library in Python with more than 500 million monthly downloads, which was acknowledged by the maintainers, showing the real-world applicability of our approach. This work represents the largest systematic evaluation of AI-driven property-based testing to date, establishing a new paradigm for scalable software auditing.

2 Agent

Our agent is built on top of Anthropic’s Claude Code, a terminal-based coding agent (Anthropic, 2025b), that allows Claude to execute bash commands and read/write files. Our agent is implemented as a natural language prompt, stored in a Markdown file, that is passed to Claude Code. As it is a prompt, our agent is easily portable to other agent implementations, like OpenAI’s Codex OpenAI (2025), or general frameworks like ReAct (Yao et al., 2022). Our main contribution is our comprehensive evaluation which demonstrates that LLM-based PBT works at scale, finding diverse bugs with few false alarms.

Our agent is built to work with Python codebases, and generates Hypothesis PBTs to test that code. It takes a single argument, which points the agent towards a particular target, either a single Python file (e.g., normalizers.py), module (e.g., numpy, scipy.signal), or function (e.g., requests.get(), json.loads()).

We developed the prompt by using industry best-practices (Anthropic, 2025a), collating high-value information from the Hypothesis documentation (Hypothesis Contributors, 2025), and iterative refinement from manual inspection of agent runs. The full prompt is in Appendix A.

Prompt Overview

The agent is given the following six instructions, paraphrased below:

Finally, we provide the agent with a short reference to Hypothesis, which shows essential patterns, some key testing principles, and links to online documentation which can be fetched if needed.

Key Design Choices

Our agent maintains a to-do list to track its progress through the 6-step cycle, as well as to keep track of, e.g., which properties it would like to test. We attempt to reduce false alarms throughout by emphasizing that proposed properties should be strongly supported by evidence, asking the agent to reflect on failing tests carefully, and to focus on a few high-value properties. Lastly, we let the agent have broad autonomy: it decides which functions to target, and decides when a failing property is truly a bug (with some direction from a rubric).

3 Experiment

We evaluate our agents ability to find bugs by curating a diverse corpus of Python packages. We combine three complementary sampling approaches to ensure comprehensive coverage:

For each package, we test the agent on the main module (e.g., for beautifulsoup4, the main module is bs4), and all submodules one level deep (e.g., numpy.linalg). In total, we ran on 933 modules.

We used Claude Opus 4.1 Anthropic (2025c) as the LLM for all agent runs. Each agent was run in an isolated virtual environment containing the package under test and its dependencies, along with Hypothesis. Agents were run in parallel, with up to concurrently. Agents had read/write permission within their working directory, could execute python and pytest bash commands, had read access to the virtual environment for source code, and internet access. The experiment was run on a RunPod Ubuntu 20.04 container with 8 vCPUs, 16 GB RAM, and 50 GB disk.

Evaluation

Definitive validation of reported bugs across such diverse libraries is difficult as it requires domain expertise in each specific codebase. First, we developed an initial scoring rubric with the goal of eliminating clear false alarms and got Claude Opus 4.1 to grade all reports. We randomly sampled bug reports from the top 80% by score. Two of the authors independently scored each bug report, using two criteria: “Is this a valid bug? If yes, would we reasonably report this to the maintainers?". Disagreements were discussed to arrive at a final consensus. Using insights from the initial rubric, we developed a final rubric (see our code) which scores bug reports out of 15, got Claude Opus 4.1 to grade all reports, and manually reviewed all reports with a score. To validate our agents with maintainers with actual domain expertise, we (manually) reported some particularly interesting bugs to their respective repositories.

4 Results

Summary statistics

We evaluated our agent across 100 Python packages covering 933 modules. The agent generated 984 bug reports, discovering issues in 786 modules (84.2%) and averaging just over one bug per module. Total agent runtime was 136.6 hours (82 min/package, 8.8 min/module) and total API cost was $5,474.20 ($54.74/package, $5.87/module, $5.56/bug report). Individual module costs ranged from $0.65 to $15.42, with a median of $5.81. Each agent run involved roughly 110 turns. In aggregate, the agents used 2.21 billion (input & output) tokens (2.37 million/module).

Manual review

Inter-rater agreement on the initial review was . After reaching consensus, 56.0% (95% CI: 42.2%, 69.8%) of reports were determined to be valid bugs, and 32.0% (95% CI: 19.1%, 44.9%) were valid and worth reporting. With the final rubric, of the top-scoring reports were valid, and of those were both valid and worth reporting.

Reported bugs

While examining reports, we selected the following particularly interesting bugs to report to the original developer. The corresponding bug reports generated by the agent are in Appendix B. They demonstrate the range of issues discoverable through our approach:

5 Discussion and Conclusion

Our evaluation demonstrates that LLM-guided property-based testing can systematically uncover bugs missed by traditional testing. With a cost of $5.56/bug report, and extrapolating from our manual grading that 56% of these are valid bugs, our agent can find bugs for $9.93/valid bug. This is an upper bound on the real-world cost, where developers with domain expertise can be more judicious with where to target the agent. The diversity of issues, spanning numerical issues to business logic issues, show the power of PBT, and the ability of agents to autonomously mine for such properties.

Limitations

The primary limitation is that we did not manually review all 984 bug reports. However, our review of a subset of reports shows that the false discovery rate has a 95% CI of [30.2%, 57.8%]. The secondary limitation is intent ambiguity: the agent cannot distinguish intentional design violations from bugs. An example of this is a bug report about LookupDict from the requests library: based on its name and the fact it subclasses from the built-in dict, our agent tested whether it exhibits dict-like behavior, which it in fact does not. The same issue was raised on GitHub in 202211 1

        https://github.com/psf/requests/issues/6238, but the maintainer replied that it was not meant to work like dict. In practice, both limitations can be mitigated by developers’ domain expertise rather than blind exhaustive testing as we did.

Comparison to Related Work

Vikram et al. (2023) study converting library documentation into Hypothesis PBTs for 40 functions across 10 popular Python libraries (e.g., numpy). With their best model and prompting strategy, only 41% of generated PBTs ran without error and passed on the implemented code, and at most 21% of documented properties were captured. In contrast, we do better by using an agentic approach, which is better able to capture more complex properties due to multiple steps. More recent work He et al. (2025) propose a different take on this problem: a “generator" LLM produces candidate code while a “tester" LLM synthesizes PBTs from the problem description. PBT failures are fed back to guide code refinement, yielding improvements over example-based test-driven development. There is also extensive work on LLMs for software testing in general: see Wang et al. (2024) for a review.

Conclusion

We presented an automated approach for bug discovery using LLM-guided property-based testing, which discovered real bugs across several popular Python libraries. The properties and failing test cases discovered show the utility of combining LLMs with PBT. While such a tool is useful for developers, a possible malicious use is autonomous discovery of vulnerabilities. As LLMs improve and get cheaper, agentic PBT could become an increasingly valuable complement to traditional testing, helping developers systematically explore code behavior and find bugs.

References

Appendix A Agent prompt

Note: some characters may not render properly in the PDF; we provide the full prompt file in our code release.

Appendix B Selected bug reports

A selection of bug reports, as written by the agent.

B.1 numpy

B.2 aws-lambda-powertools

B.3 cloudformation-cli-java-plugin

B.4 tokenizers

B.5 python-dateutil

NeurIPS Paper Checklist