How to Build an AI Software Factory

Firecrawl's synthesis of the published "AI software factory" architectures — Stripe Minions, Spotify Honk, Shopify River, Ramp Inspect, Faire, Uber uReview — distilled into a five-stage control plane around a coding agent: intake (is this worth starting), isolation (where it runs), tools (what it can reach), verification (is it right), merge gate (who is accountable). The thesis is economic — generation scales with spend and review does not, so the factory exists to absorb agent output — and the best feature is the numbers section that does not flatter the idea. It is also, transparently, a Firecrawl product document: nearly every stage resolves to a Firecrawl call.

Key Quotes

"AI changes the economics of code production. One person with good judgment and a phone can generate PRs faster than a team can review them."

Stephen Toub, writing up the flight where he opened nine dotnet/runtime PRs from 35,000 feet and merged seven. The load-bearing word is review. The whole five-stage architecture exists because of this asymmetry: every stage after intake is some company's answer to "how do we absorb what the agent produced?"

"On a business level, the moat of software companies will shift from 'the code they wrote', to the 'means of production' of that code. The alpha is in your factory."

Guillermo Rauch, on Vercel open-sourcing its own reference platform for cloud coding agents. The strategic claim the article adopts as its frame — the same move Cloud Software Factories makes with an efficiency equation and The Dark Factory is a DOT File makes with the pipeline-as-artifact argument. Code is becoming commodity; the factory is becoming the asset.

"Miss the third and you have not built a software factory. You have built a machine that generates review work faster than you can absorb it."

The definitional line of the piece. Three properties make a factory: queue-driven intake, disposable environments, verification before review. The third is the one teams skip, and skipping it produces the failure mode the article is organized around — a machine that mints review debt faster than humans can service it. This is the Guardrails and Feedback Loops principle (deterministic checks before human attention) stated as an admission criterion.

"Self-reported confidence is not a quality signal. A second model asking 'is this comment worth a human's time' is."

Faire's negative result, and the most valuable single data point in the article. Filtering faire-review comments by the model's own confidence score threw away 65% of comments at the strictest gate and moved acceptance by 3% — with a 0.93-confidence comment dismissed sitting beside a 0.35-confidence comment accepted and fixed. The obvious quality lever is dead. What lifted acceptance to 73% was better context about the change, better targeting of where a comment is worth making, and a second model grading worth-a-human's-time.

"A failed PR generation is an annoyance. A PR that fails CI is a burden on an engineer. A PR that passes CI and is functionally wrong erodes trust in the whole system. Design toward the first failure and away from the third."

Spotify's failure-mode ranking. Most verification writing stops at "run the tests"; this is a company publishing its error economics. Cheap failures are fine — trust-eroding failures are the design target, which is why the LLM judge vetoes ~25% of agent sessions before CI ever burns a cycle.

"Writing code is now easy; the real work is reviewing it."

Steve Ruiz, when tldraw paused external contributions. Paired with curl taking all of July 2026 off from vulnerability reports — "possibly our best project decision in a long while" — and noting the flood hasn't slowed on the other security teams Daniel Stenberg sits on, this is the externality chapter: private factories export their review burden onto open source, and the only intervention that worked was closing intake, which is a gate argument.

Key Themes

#concept #pattern #tool #person

• #pattern — Five gated stages. Intake (Sentry's Seer scores errors for actionability; Shopify's River works only in public channels — "every conversation is therefore searchable"; Microsoft's size filter, since 1–50-line agent PRs succeed 76–80% while performance work lands at 54.5%), isolation, tools, verification, merge gate. Stage order matters more than tooling: expensive stages sit at the end, and each gate stops work from reaching the next.
• #pattern — The isolation ladder. Worktrees (2–5 agents, one machine) → containers (conflicting deps, untrusted changes) → cloud sandboxes (fleet concurrency; Stripe's pre-warmed devboxes, Ramp on Modal, Spotify on Kubernetes). The part every worktree guide skips: a fresh checkout has no .env, no node_modules, no free port — that is what actually breaks at agent four, which is why teams conclude worktrees "don't scale" when they hit an unconfigured checkout.
• #pattern — Tools are fleet-wide, not per-agent. Stripe's Toolshed: ~500 internal tools behind one MCP server, with destructive actions blocked. Cloudflare: an internal MCP Portal and generated AGENTS.md across 3,900+ repositories. OpenAI's Codex team runs a ~1M-line repo on a ~100-line AGENTS.md that "serves primarily as a map." Microsoft: instructions plus firewall config moved agent success on dotnet/runtime from 41.7% to a sustained 71% — preparation worth more than any model upgrade in the same period.
• #pattern — Verification is nested loops, cheapest first. Inner loop: deterministic verifiers, no model. Judge: an LLM checking whether the diff stayed in scope — Spotify's vetoes ~25% of sessions, and about half of those are recoverable by course-correcting rather than discarding. Outer loop: CI, with Stripe capping agents at two runs before a human, because at 3M+ tests a CI cycle is the expensive resource. Checks ordered by unit cost: 5-second lint before 2-credit lookups before CI cycles.
• #pattern — Merge gates tiered by blast radius. Mechanical/reversible → automated checks and auto-merge; scoped fix → one review; agent-authored non-trivial → two reviews (Faire); domain-sensitive (auth, payments, crypto, data deletion) → named domain owner regardless of author. Faire's sequencing rules — no code-owner review until a first review exists; ready-for-review only when the agent requests it — stop agents from pulling domain experts into unfinished work. Mastra separates three independent approvals: accept the task, approve the plan, merge the PR.
• #concept — Attribution policy. The Linux kernel's 2026 rule: agents must not add Signed-off-by: — a legal assertion a model cannot make — and must add Assisted-by: AGENT_NAME:MODEL_VERSION instead. GitHub's Agent HQ manages agent identity the way it manages human developers. "Either is defensible. Silence is not," because six months later nobody can answer which changes came from where.
• #concept — The dedup gate that fails open. At Stripe's 1,300 merged agent PRs a week, spawning agents onto already-solved problems is exactly the waste a factory exists to remove. A gate written if not results: spawn_agent() cannot tell "nobody has reported this" from "we have no coverage of that repo" — both return 200 with zero results. Reading repos[0].indexed == false as unknown rather than clear is the one-line fix. A general lesson for gates built on retrieval: absence of evidence is not evidence of absence.
• #concept — The honest numbers. Agent PRs merge less often (67.9% vs 87.1%; fully autonomous only 55.1%, human-committed 86.2%; revert rates flat at 0.6% vs 0.8%). METR: roughly half of 296 SWE-bench-passing AI PRs would not merge — a ~24-point grader-to-maintainer gap that did not shrink from mid-2024 to mid-2025. DORA: throughput and instability rise together; ~39% first-year ROI only through a J-curve dip attributed partly to a "verification tax"; 35–40% gains on simple tasks, often under 10% on legacy code. GitClear: duplicated blocks up eightfold, refactoring's share down from ~25% to under 10%, churn doubled to 7.1%.
• #concept — The proven workload is migration. Spotify's fleet-wide changes, Airbnb's test migration (3,500 files in six weeks, 75% in the first four hours), Amazon's Java upgrades — "all the same shape, all mechanical, all at a scale humans resent." The factory is a maintenance machine, and Microsoft's gradient ("excellent at implementing well-specified changes... relatively poor at architecting solutions") draws its ceiling.
• #tool — The published fleet. Stripe Minions (1,300+ merged agent PRs/week), Spotify Honk + Fleetshift (shipped 2023, two years before there was an agent to put in it), Shopify River (59,918 sessions, 3,536 co-authored PRs merged in one 30-day window — "about one in eight" merged PRs), Ramp Inspect (~30% of merged PRs, no mandate), Uber uReview (>90% of ~65,000 weekly diffs; 65% of its comments addressed in the same changeset vs 51% for human reviewers), Sentry Seer, Faire faire-review.
• #person — The chorus. Stephen Toub (economics), Guillermo Rauch (strategy), Tobi Lütke (org design via public channels), Addy Osmani ("a software factory is harnessing loops at scale"), Steve Ruiz and Daniel Stenberg (the open-source externality), OpenAI's Codex team (the honest confession that before automating the loop, "our team used to spend every Friday (20% of the week) cleaning up 'AI slop.' Unsurprisingly, that didn't scale").

Critical Analysis

The best-cited document in the factory genre. Every claim in the load-bearing sections is anchored to a named, published system with numbers attached, and the stage-comparison table (Stripe/Spotify/Shopify/Ramp across all five stages) is genuine synthesis work — the same skeleton extracted from four architectures that each use different vocabulary. Where Cloud Software Factories is a vendor's abstraction, this is the same skeleton with the receipts stapled to it. The counterweight section is what earns the piece its credibility: agent merge rates below humans, the METR grader gap, GitClear's duplication data, and DORA's instability findings are all arguments against the product being sold, printed inside the sales document.

It is also an ad — and mostly honest about it. The dedup gate is the Firecrawl developer index, live-docs verification is firecrawl_developer_search, and the safety story is prompt-injection detection on scraped pages. What keeps it readable is that the vendor parts carry their own negative numbers: DevDex answers correctly in its top ten 63% of the time, presented with "roughly a third of queries still miss. It narrows the search, it does not end it." A vendor publishing its own miss rate is rare enough to deserve note. The one place the seam shows is the migration pitch: "what to migrate to is an external fact" is true, but the article quietly converts every factory's hard problem into a search problem that happens to be Firecrawl's product.

Gates are proven; fleets are aspirational. The most useful sentence for a practitioner is the last one: "Pick the stage where your agents currently produce the most rework, and put a gate there. Not a fleet, not an orchestrator, one gate." This inverts the genre's build-the-platform pitch: Spotify shipped Fleetshift two years before it had an agent to put in it, and every honest failure story in the piece (OpenAI's slop Fridays, Faire's flat reviewer count, the review-capacity math: 16.5 comments per merged agent PR vs 12.4 human, with 36% of feedback from the top two reviewers) is what a company learned by hitting the wall, not by buying the platform. Read next to Harness Engineering is not Enough's counter-argument — that no factory infrastructure compensates for code never built to be maintained, because architecture costs play out beyond any training or gate horizon — the fair summary is: the gates are empirically supported, the full lights-off fleet is still a bet.

Evidence against the end of code review, from the factories themselves. The End of Code Review argues mandatory human review is no longer defensible. This survey is the current counter-record: every published factory keeps a human at the merge gate, Faire doubles the requirement on agent-authored work, and the kernel refuses to let a model make the legal assertion that Signed-off-by: carries. The one exception — OpenAI's Codex team pushing "almost all review effort towards being handled agent-to-agent" on a greenfield internal codebase with no external users, built by the team that makes the agent — proves the rule by its extremity. Monperrus's bottleneck math is confirmed exactly (review is the fixed-capacity stage); his conclusion is simply not what anyone outside a greenfield lab ships. 6 Learnings from 12,000 Agentic Code Reviews shows where agent-to-agent review does work — but as worst-of-seven gating lanes inside a harness, not as the accountability layer.

What the numbers quietly argue. "Passing tests is not the same as being mergeable" (METR) and "the last 10% might last a very long time and cost many tokens" are the two sentences a factory builder should tape to the monitor. Together with Microsoft's 41.7%→71% prep result, they say the leverage is upstream of the model — in instructions, scoping, and intake filtering — and downstream of generation, in verification. The agent is the cheap part, as the TL;DR admits; almost the entire article is about everything else.

Related

• Cloud Software Factories — Lloyd's vendor blueprint and this piece share one skeleton (centralized runtime, orchestration, measurement; humans at decision points). This source strengthens the blueprint by attaching published architectures and unflattering numbers to each layer, and complicates Lloyd's build-vs-buy lean with evidence that the proven factories were assembled gate-by-gate in-house after hitting specific failures — "start with one gate" is the DIY-first inversion of "most companies should not build factory infra."
• Loop Engineering — The article quotes Osmani's own compression of it: "a software factory is harnessing loops at scale." It strengthens the loop taxonomy by showing its components — worktrees, skills, connectors, subagents, on-disk state — reappearing unchanged as factory stages (isolation, tools, verification, the session log), and nuances it with the review-capacity ceiling that single-operator loop engineering never hits: at fleet volume, the loops themselves generate the bottleneck.
• The End of Code Review — This survey complicates Monperrus's thesis directly rather than refuting it. The bottleneck math is confirmed (16.5 review comments per agent PR; 36% of feedback from two people), but the merge-gate evidence — Faire's two-review rule, the kernel's Signed-off-by: ban, OpenAI's greenfield outlier — shows every published factory keeping humans exactly where the paper says they cannot stay: at the accountability decision.
• Agentic Code Review — Osmani's "writing got cheap but understanding didn't" is this article's organizing asymmetry stated six words shorter. Faire's confidence-score negative result and Uber's precision-beats-volume data give his heterogeneity and human-on-the-loop prescriptions their concrete published instrumentation, and his warning that agents "will weaken CI to make themselves pass" is the reason factory verification starts with deterministic loops a model cannot talk down.

Also related, as branches of the same family: Self-Hosted Sandboxed Agentic Software Factory is the solo, blast-radius-contained version of the same five-stage shape; Don't Fear the Dark Factory makes the maximalist case this article's numbers keep deflating; Uber — Agentic Engineering Shift is the uReview operator's own account of the same system; What a User Story Actually Costs in a Dark Code Factory prices the factory's unit economics from inside one.

Sources: raw/ai-software-factory, How to Build an AI Software Factory: Agents That Open, Review, and Merge PRs Last updated: 2026-09-13