Agentic Manual Testing
Simon Willison's guide to making coding agents verify their own output by actually running it. The core argument: automated tests are necessary but insufficient — you need the agent to manually exercise the code (via python -c, curl, or browser automation) because tests can pass while the server crashes on startup. A chapter from his Agentic Engineering Patterns guide, built around two of his own tools (Rodney for browser control, Showboat for documenting the testing flow).
Key Quotes
"The defining characteristic of a coding agent is that it can execute the code that it writes."
This is Willison's crispest formulation of what separates a coding agent from an LLM that just spits out code. Execution closes the loop. Without it, you're proofreading syntax, not verifying behavior.
"Never assume that code generated by an LLM works until that code has been executed."
The first principle. Obvious stated, ignored constantly. Every practitioner who's shipped agent-generated code has a story about the thing that looked perfect and failed immediately when run. This isn't paranoia — it's the empirical base rate.
"Automated tests are no replacement for manual testing."
Willison's most contrarian claim in a testing ecosystem that treats automated coverage as the gold standard. Tests can pass while the feature is visibly broken — server won't start, UI element missing, API returns 200 with wrong data. The agent needs to see it work.
"I like to see a feature working with my own eye before I land it in a release."
The human parallel. If you wouldn't ship code you haven't seen run, don't let your agent ship code it hasn't seen run. This isn't Luddism — it's the same instinct that makes you refresh the browser after git push.
"The
execcommand is the most important of these, because it captures a command along with the resulting output."
About Showboat's command-capture tool. The deeper point: LLMs hallucinate results they expected to see. Capturing actual stdout/stderr is the antidote. Without it, you get Write Only Code dressed up in a README.
Key Themes
Execution as the defining trait
Willison draws a bright line: LLMs that output code vs. coding agents that output and execute code. The ability to run what it writes is what makes an agent useful rather than merely impressive. This aligns with the Don't Fear the Dark Factory framing — the dark factory is a validation problem, not a generation problem, and execution-is-verification is the simplest validation loop.
Tool design for agent-native testing
The article contains embedded tool-design wisdom. Willison's approach to Rodney and Showboat is deliberate: (1) --help output that teaches the agent everything it needs, (2) uvx for zero-friction first install, (3) screenshot capabilities that leverage the agent's own vision. This is 10 Principles for Agent-Native CLIs applied to testing tools — design for the agent first, humans benefit from the same clarity.
Compare with surf-cli (browser automation via CLI and Unix sockets) and Browser Use (AI browser automation with anti-detection). Rodney is the simplest of the three: CDP, no MCP, one binary.
The fragility tradeoff
Willison's take on browser test flakiness is pragmatic rather than ideological: don't avoid browser tests because they're fragile — let agents maintain them. The maintenance cost that made Selenium tests a running joke is radically different when you're not the one doing the maintenance. This is a specific instance of Compound Engineering's general principle: when you can't trust the output, add a system, not manual review.
Showboat as documentation-by-execution
Showboat solves a specific failure mode: agents that write documentation describing what should have happened rather than what did happen. The exec command captures reality. This connects to If AI Is Doing the Investigation, Version the Investigation — both are about making the agent's actual observed behavior the durable artifact, not its retrospective summary.
Critical Analysis
This is a short chapter (part of a larger guide), but it's denser than it looks. The practical techniques — python -c, /tmp demo files, curl exploration — are the kind of thing experienced practitioners do instinctively but rarely write down. Willison writing them down is valuable precisely because they're obvious: obvious things are the first to be forgotten when documenting workflows.
What it gets right: The multi-language coverage (Python, compiled languages, web APIs, browser UIs) makes this a reference you can hand to an agent. The uvx rodney --help pattern — where the tool teaches itself to the agent — is genuinely clever and under-copied. The Showboat concept addresses a real failure mode (agents hallucinating test results) that I haven't seen addressed elsewhere.
What it skips: The article assumes the agent is self-verifying in a single session. Multi-session testing — where tests written today catch regressions next week — gets a nod (red/green TDD for fixes) but isn't the focus. There's no discussion of context-window economics when the agent's testing loop consumes tokens. And "have the agent maintain the tests" as the answer to flakiness assumes the agent won't also write flaky tests — which, empirically, they do (see AI Coding Tools Create More Bugs Than They Fix).
The Willison pattern: Between this, Designing Agentic Loops, and 2025 in LLMs, a distinct Willison methodology is emerging: shell commands over MCP, self-documenting tools via --help, execution-as-verification, and a preference for pragmatic heuristics over heavyweight frameworks. It's the most coherent "school" of agentic development I've seen outside of Anthropic's official guidance. If you had to pick one person's approach to copy, his would be it.
What's missing from the ecosystem: A testing framework designed for agent use rather than human use. Playwright was built for human QA engineers. Rodney repurposes CDP for agents but it's a thin wrapper. What we need is something that combines: (1) the agent writes the test in natural language, (2) the framework executes it against the actual UI, (3) visual diffs + DOM snapshots are captured automatically, (4) the agent reviews its own results and iterates. Showboat + Rodney is a prototype of this, but it's not productized.
Cross-Links
- Designing Agentic Loops — same author, same guide series. The meta-skill this chapter is an instance of
- Teaching Claude to QA a Mobile App — concrete field report of agentic manual testing on mobile, with CDP on Android and pain on iOS
- Feedback Loop is All You Need — tests as the feedback mechanism that makes agents converge
- Harness Engineering — the theoretical framework: manual testing as computational feedback
- Don't Fear the Dark Factory — the validation problem vs. generation problem; this article is a validation answer
- Compound Engineering — "add a system, not manual review" — letting agents maintain browser tests is a compound-engineering move
- surf-cli — browser automation for agents via CLI; same domain, different implementation philosophy
- Browser Use — AI browser automation with anti-detection; complements Rodney's CDP approach
- 10 Principles for Agent-Native CLIs — Rodney and Showboat are textbook examples of agent-native CLI design
- 2025 in LLMs — same author's annual survey; broader context for his thinking
- Spec-Driven Development — tests, specs, and code as a triangle; manual testing verifies the spec, not just the code
- Slowing the Fuck Down — manual testing as deliberate friction; the opposite of push-and-pray
- Write Only Code — what you get when you skip manual testing
- Maestro (UI Testing) — end-to-end UI testing for mobile and web; the non-agent alternative
- If AI Is Doing the Investigation, Version the Investigation — Showboat's
execcommand as investigation versioning - AI Coding Tools Create More Bugs Than They Fix — the counterpoint: agents that write flaky tests create maintenance debt, not maintenance relief
Sources: Agentic manual testing Last updated: 2026-05-15