How to Build an AI Software Factory: Agents That Open, Review, and Merge PRs

Firecrawl's synthesis-to-blueprint of the AI software factory: the system around a coding agent rather than the agent itself. Work arrives from a queue, agents run in disposable isolated workspaces, verification runs before any human looks, and a person sits at an explicit merge gate. The frame is Stephen Toub opening nine dotnet/runtime PRs from an airplane (seven merged): one person can now saturate a team's review capacity, so the design problem stopped being producing agent output and became absorbing it. A factory differs from a pile of scripts by three properties — queue-driven (not prompt-driven) intake, disposable environments, and verification before review; miss the third and "you have built a machine that generates review work faster than you can absorb it."

The five stages, each anchored to a published architecture. Intake filters what is worth starting: Sentry's Seer scores incoming errors for actionability, Shopify's River works only in public channels ("every conversation is therefore searchable"), and Microsoft's data argues for a size filter (1–50-line agent PRs succeed 76–80%, performance work 54.5%). Isolation climbs worktrees → containers → cloud sandboxes (Stripe's ~10-second pre-warmed devboxes; the trap that actually breaks at agent four is the fresh checkout with no .env, node_modules, or free port). Tools give the agent hands: Stripe's Toolshed serves ~500 internal tools over one MCP server, Cloudflare generates AGENTS.md across 3,900+ repos, and Microsoft moved agent success from 41.7% to 71% with instructions and firewall config alone. Verification is nested loops, cheapest first: deterministic inner loop, an LLM judge that vetoes ~25% of Spotify agent sessions, CI outer loop (Stripe caps agents at two CI runs). Faire's negative result — filtering review comments by the model's self-reported confidence discarded 65% of comments and moved acceptance 3% — while a second model asking "is this comment worth a human's time" is what actually worked; Uber's uReview takes precision over volume across >90% of ~65,000 weekly diffs. The merge gate tiers by blast radius: Faire requires two reviews on agent-authored PRs; the Linux kernel bans agents from Signed-off-by: in favor of Assisted-by: trailers; OpenAI's internal Codex project reviewing agent-to-agent with humans optional is the lone outlier.

The numbers section is unusually honest for vendor content: agent PRs merge less often than human ones (67.9% vs 87.1% on dotnet/runtime; fully autonomous only 55.1%), METR found roughly half of 296 SWE-bench-passing AI PRs would not be merged by maintainers (a ~24-point gap that did not shrink from 2024 to 2025), DORA finds throughput and delivery instability rising together through a "verification tax" J-curve, and GitClear charts an eightfold rise in duplicated code blocks. Review capacity is the recurring break point — generation scales with spend, review does not — and the proven workload is mechanical migration at scale (Airbnb's 3,500 test files in six weeks) rather than greenfield design. The closing advice: start with one gate, not a fleet. Every stage also, transparently, routes through a Firecrawl product: the developer-index dedup gate, live-docs search, and prompt-injection detection on scraped pages.