On a Year of Multi-Model Development
Jason A. Hoffman's field report on running Claude Code, OpenAI's Codex CLI, and Google's Gemini CLI together through a shared MCP layer for a year. The piece is equal parts practitioner journal, model taxonomy, and argument that the human specifier -- not the model -- is the bottleneck.
Key Quotes
"Plausible output that looks like understanding but is pattern completion."
All five model configurations independently identified this as their core vulnerability when prompted for honest self-assessment. The convergence is more interesting than any individual answer.
"The specification is the bottleneck, and the specification comes from the human."
The thesis statement. Hoffman's acceleration numbers (25-71x) only make sense if the human brings domain knowledge, precise requirements, and judgment about which model to use for what.
Delegates "from knowledge rather than ignorance."
The sharpest line in the piece. The effective multi-model operator isn't someone who can't do the work -- it's someone who can, and chooses to direct models instead.
Key Themes
The construction-trade taxonomy. Codex as drywall contractor (fast, unquestioning, spec-dependent), Gemini as surveyor (broad coverage, misses buried constraints), Claude as architect (handles complexity, risks overdesign and eloquent concealment). These aren't benchmarks -- they're operational heuristics for task routing. The characterizations are more useful than any leaderboard because they name the failure modes, not just the capabilities.
Human-driven routing, not automated routing. There is no model router. The human decides based on task shape, failure mode awareness, and domain knowledge. This is the opposite of what most orchestration frameworks assume -- Cord, maestro, and Scaling Long-Running Agents all route programmatically. Hoffman's argument is that the routing decision itself requires judgment that models can't provide.
Acceleration varies by 28x across task types. Mechanical work hits ~12x. Strategic memos hit ~2.5x. The variation matters more than the headline number. Organizations that measure "AI productivity" as a single metric will systematically misallocate. The 71x project-level number came from a domain with high pattern reuse; the 25x came from novel infrastructure work. This is empirical confirmation of what The Claude C Compiler found theoretically: known abstractions accelerate dramatically, novel design barely budges.
Deterministic output over LLM generation for critical documents. Financial reports should be generated by code, not by language models, so humans can trace data sourcing. This echoes Smart Models, Dumb Pipes's separation: models own judgment, deterministic systems own execution and audit trails.
Self-assessment convergence. When asked to evaluate themselves honestly, all models said the same thing and then recommended themselves for the most important work. Codex was the exception -- it web-searched for benchmarks instead of introspecting. The pattern-matching critique applies to the models' self-evaluation too: they generated plausible self-awareness that was itself pattern completion.
Critical Analysis
Hoffman's construction-trade metaphors are the piece's strongest contribution. They compress months of operational experience into immediately actionable heuristics. When someone asks "which model should I use?", the answer is no longer a benchmark comparison -- it's "what kind of contractor do you need right now?"
The acceleration numbers are the most granular published data I've seen on multi-model development. The 2.5x for strategic memos is the honest number that most AI productivity claims omit. It quietly confirms Cognitive Debt's thesis: the closer work gets to novel thinking, the less AI accelerates it. The 71x headline is real but misleading without the task-type breakdown.
The weakest part is what's missing: failure stories. Hoffman describes what each model is good at but doesn't detail the projects where multi-model coordination produced worse outcomes than a single model would have. Every orchestration system has coordination overhead. At what point does switching between three CLIs cost more in context loss than it gains in specialization? The piece implies the answer is "never, if you're skilled enough," which is the kind of unfalsifiable claim that should make practitioners suspicious.
The "delegates from knowledge" framing is powerful but carries a quiet elitism. It implies that the productivity gap between a 71x developer and a 2x developer is primarily about the human's existing domain expertise. This is probably true, and it's uncomfortable: AI tools amplify existing human capital rather than democratizing capability. Radical Accountability arrives at the same destination from a different angle -- taste and judgment are what remain when execution is cheap.
The MCP-as-shared-bus architecture is described but not examined. Running three competing models through a shared context layer raises questions about context contamination, conflicting conventions, and debugging difficulty that Hoffman doesn't address. The closest analogue in the wiki is Agent Orchestration's finding that flat coordination fails -- but Hoffman's setup works because there's a single human coordinator, not because the architecture is inherently sound.
Cross-Links
- Specifications as the Product -- Hoffman's "specification is the bottleneck" is the practitioner's version of this synthesis page's core thesis
- Agent Orchestration -- Hoffman's human-routed multi-model setup is the manual version of what orchestration frameworks automate
- Smart Models, Dumb Pipes -- The deterministic-output-for-critical-documents argument maps directly onto McCormick's separation of judgment and execution
- Radical Accountability -- Both argue that when execution is cheap, human judgment is the remaining differentiator
- Cognitive Debt -- The 2.5x for strategic memos confirms: comprehension-heavy work resists acceleration
- The Claude C Compiler -- Lattner's "known abstractions accelerate, novel design doesn't" matches Hoffman's acceleration variance
- Compound Engineering -- Hoffman's pattern reuse driving higher acceleration is the compound loop in action
- Five Levels from Spicy Autocomplete to the Dark Software Factory -- Hoffman operates at Level 4, with multi-model coordination as the distinguishing feature
- acceleration-flow -- Hoffman's measured numbers are the antidote to Marc's unmeasured acceleration anxiety
- Scaling Long-Running Agents -- Programmatic routing vs. Hoffman's human routing; different solutions to the same task-allocation problem
- Fresh Eyes -- Hoffman's model-switching when stuck is the manual version of cross-model review
Sources: On a Year of Multi-Model Assisted Development Last updated: 2026-05-14