Agent Orchestration

topic · 114 sources

Agent orchestration is three problems that keep being conflated: agent-to-agent coordination, human-to-agent coordination, and state persistence across both. On the first, the evidence is unusually unanimous — hierarchy beats flat coordination, rediscovered independently at every scale from a fifteen-line bash loop to Uber's platform organisation. On the second, task boards and dependency graphs have won as the coordination surface, and the newest work is promoting that surface into a full organisational runtime: budgets, org charts, approval gates, audit trails. On the third, the field has settled on file-based memory, worktree isolation, atomic task claiming, and disposable sessions over durable state. The theoretical through-line, supplied by Multi-Agent AI Systems Are Organizations, is that these systems are organisations rather than architectures, inheriting task division, allocation, information provision and reward distribution by construction. What nobody has solved is cross-agent learning: agents do not learn from each other's discoveries.

The Argument

Flat coordination fails; hierarchy works

This is the most robust finding in the evidence, and it has only strengthened as more of it arrived. Scaling Long-Running Agents documents Cursor's experiments: flat self-coordination with shared files and locking reproduced every pathology of a badly-managed human team — agents held locks indefinitely, avoided difficult tasks, and twenty agents slowed to the effective throughput of two or three. The fix was planner/worker/judge separation: planners map the codebase and generate tasks recursively, workers execute without coordinating with each other, and judge agents evaluate completion. How Hightouch Built Their Long-Running Agent Harness reached the same conclusion independently, finding that DAG-heavy frameworks failed on open-ended marketing tasks and that separating planning from execution — with an update_plan tool — was essential. Agent Swarm Model Economics supplies the economic confirmation: Cursor's flat swarm burned $10,565 in API spend and thrashed, producing far more commits, merge conflicts and lines of code, while the tree-structured hybrid cost $1,339 and converged.

The hierarchy takes different shapes depending on the task, but the shapes are consistent. maestro uses PM/Architect/Coder roles, the Architect reviewing but never writing code, so the role that designs is not the role that implements. Gas Town's Agent Patterns uses Mayor/Polecats/Witness/Refinery, with the Mayor as the single human interface and the Refinery resolving merge conflicts. Cord gives agents five primitives — spawn, fork, ask, complete, read_tree — to build task trees at runtime rather than predefining them; the spawn-versus-fork distinction (clean slate versus sibling context injection) is its key contribution. The Grid shows the same separation encoded as identity rather than code: twelve named programs, each a role identity rather than a different model vendor, whose file-based identity — temperament dials, a soul file, boundaries — shapes how they act. Pi Subagents adds the same delegation to the Pi agent, with capability ceilings so a child can be restricted below its parent.

Building an Advanced Agentic Harness gives the cleanest decomposition of why the split works: Planner/Worker/Critic separation turns one confused prompt into three narrow agents with single contracts, each testable and swappable, and the orchestrator becomes deliberately thin. TDD Coordinator (Corazonn) implements it as a Claude Code slash command with a mandatory Rule of Two — every work product, test and implementation alike, reviewed by a different agent before it counts as done. Mole pushes the judge out of the peer role for deep research, treating planner/executor/verifier as pipeline stages with independently attributable spend. The enterprise evidence confirms the pattern scales: Orchestrating AI Code Review at Scale runs a coordinator that fans out to up to seven specialist reviewers and then deduplicates and judges their output, and Dynamic Workflows in Claude Code is Anthropic's first-party version, isolating subagents in their own context windows to counter the agentic laziness, self-preferential bias and goal drift that emerge when planning and execution share a context.

At the bottom end the autonomous loop — Ralph, Fleet of Agents (sermakarevich), Fleet Supervisor (sermakarevich) — is a degenerate hierarchy with the human as planner. Ralph is a bash while loop calling claude -p against a fix plan on disk; its power is that nothing in the agent changed. Fleet of Agents (sermakarevich) builds it up in five steps, from a single agent writing task context to disk through an on-disk queue, a bash loop, parallel agents with atomic task claiming, and finally agents that can stop and ask questions when blocked. Fleet Supervisor (sermakarevich) is the production version — a Python supervisor claiming from a central queue, with worktree isolation, context-pressure detection, and MCP-based human-in-the-loop. What Ralph Wiggum Loops Are Missing maps the graduation path from freeform markdown to structured dependency tracking, arguing Ralph and richer coordinators are stages on one learning curve, not competitors. Loop Engineering generalises the whole thing: design systems that prompt agents autonomously rather than prompting agents directly. Local Deep Research applies the fan-out to research, spawning parallel subagents that each run their own search tools and deduplicate results in a shared collector.

Why hierarchy wins is no longer mysterious. Multi-Agent AI Systems Are Organizations explains it in the language of organisation science: whenever multiple bounded agents pursue system-level goals, task division, allocation, information provision and reward distribution all have to be solved, and hierarchy is one of the few answers humans keep rediscovering. The same paper adds a twist that matters throughout this page — artificial agents lack intrinsic motivation, so specification gaming, not shirking, is the dominant failure mode.

The coordination surface between humans and agents

Kanban boards have become the default human-agent interface, but the evidence increasingly suggests they are a local maximum on the way to something more structural.

Managing Agents via Kanban Boards is the clearest articulation: a Notion board whose "blocked" checkbox turns a card red when an agent needs input, with a CLI tool that blocks execution until the human responds. It emerged from use, not design — "malleable software." ralph-ban implements the same idea as a vim-navigable TUI with SQLite backing. weft is the web-native version, with every state-mutating action requiring human approval. vibe-kanban tried to be the comprehensive solution and is now sunsetting — a reminder that agent-management middleware is a thin layer and the value accrues to agents and platforms. Mission Control — Bhanu's 10-Agent Squad on OpenClaw is the best-published production reference: ten specialised agents with layered file memory, staggered 15-minute heartbeats, a six-stage board, and the hard-won advice to start with two or three agents and build a UI above three.

The board is a local maximum because it flattens dependency. workgraph makes the dependency graph the durable artifact — "agents can come and go, the graph remains" — with claims, handoffs, evidence, failures and human judgment as first-class operations. poietic wraps a methodology ("graph work") and a legal form (a public-benefit corporation) around the same idea, the graph persisting while agents stay ephemeral. speedrift-ecosystem takes the graph idea to a central control plane that observes task graphs across repositories, detects drift, and dispatches corrective actions under policy.

Beyond the graph, the direction of travel is toward running an organisation, not running agents. Paperclip is the fullest expression: an org chart, per-agent budgets with hard stops, goal-to-task context inheritance, a ticket system with an immutable audit log, and board-level governance — deliberately not an agent framework; "if you have one agent you probably don't need Paperclip; if you have twenty, you definitely do." Paca makes agents first-class Scrum teammates sitting on the board beside humans, with a WASM plugin sandbox and per-conversation Docker isolation. Cloud Software Factories makes the same argument from the engineering-leadership side: move agents off laptops onto a standardised cloud runtime, orchestrate them from the ticketing system, and measure shipped product per token, because interactive agents have hit a wall of ungovernable sprawl. Uber — Agentic Engineering Shift is the enterprise-scale confirmation — background agents, a PR inbox, and a large-scale change platform running migrations across hundreds of PRs — with the recurring finding that expensive models plan and cheap models execute.

Gas Town After 10,000 Hours of Claude Code is the standing objection to all of this: full delegation costs the operator visibility and agency. "It feels like deferring everything to agents, and I get almost no visibility of what's going on." The board is the compromise that keeps the human in the loop; the fully autonomous runtime is the version that removes them.

And all of it is single-player. Zero Alignment names the gap: software is a team sport, but orchestration tools treat it as a solo activity with AI assistants — one developer with two dozen agents and no alignment produces chaos regardless of the orchestration pattern. The first answers are appearing. QM (Multiplayer Agent Harness) gives every employee an isolated workspace plus shared Slack-channel collaboration. Munder Difflin — Clones of You, Not a Shared Bot runs a clone of each individual peer-to-peer, so a teammate's clone can ask yours how the billing service works at 3am. Moltbook is the extreme case — agents talking to agents on a social network, heartbeat-driven, with prompt injection as the live hazard. None of them yet shifts the dominant interface from a single human steering to a team steering together.

Context and state management is orchestration

The same primitives recur across every source. They are not implementation details; they are the orchestration itself.

File-based memory. Agents write state to disk because the model forgets everything between runs. Mission Control — Bhanu's 10-Agent Squad on OpenClaw layers five kinds: session, working (WORKING.md), daily log, long-term (MEMORY.md), and identity (SOUL.md and AGENTS.md). Ralph keeps a fix plan plus an AGENT.md the agent updates with discovered best practices. Fleet of Agents (sermakarevich) writes TASK.md, PLAN_AND_PROGRESS.md and FINDINGS.md per task — "you stop being the agent's memory." Loop Engineering puts it bluntly: the model forgets between runs, so the memory has to be on disk.

Worktree isolation. Running agents in one directory collides even when they edit different files, because a shared git index lets commits pick up each other's uncommitted changes. Parallel Coding Agents Guide makes the case for one git worktree per agent. Orca, Agent of Empires, Dorothy and Fleet Supervisor (sermakarevich) all implement it.

Ephemeral sessions over persistent state. Gas Town's Agent Patterns describes sessions that are "disposable by design" — identity and tasks in Git, sessions killed and respawned, with "seancing" to let a new agent ask a predecessor about unfinished work. maestro terminates coders between stories. The agent process is cheap and replaceable; the task state is the durable artifact. The complement is durability through checkpoints: Sidekick — Persistent Worker Agent Skill has a worker that must survive crashes write a STATE.md every five minutes, bounding data loss.

Atomic task claiming. When many agents pull from one queue, races are inevitable. Fleet of Agents (sermakarevich) and Fleet Supervisor (sermakarevich) solve it with beads, a SQLite task database whose bd update --claim atomically assigns a task to a claimer; the supervisor's spawn loop stays dumb while atomicity lives in the queue.

Deterministic scaffolding beats agentic decomposition where the task is known

This is the newest and most consequential thread in the evidence. Against the Cord position — let agents discover the task tree at runtime — a second camp argues you should write the decomposition down once, deterministically, and use the agent only where intelligence is genuinely needed.

Reducing Token Spend with Deterministic Workflows is the sharpest statement. A code-review skill whose coordinator dispatched 23 sub-agents burned roughly 4.5M tokens and twelve minutes per run; replacing the coordinator with deterministic workflow code that calls agents only where intelligence is required cut it to roughly 500K tokens, three agents and half the time, and flipped the skill from failing open to failing closed. The principle: "You use the Agent to build the program that minimizes the need for the Agent itself."

The same bet shows up as a DSL with compiler-grade tooling in Tracker and Dippin (Language) — a static DAG defined up front so it can be validated, simulated, linted and cost-estimated before a single token is spent; parenthesised condition expressions are rejected at parse time rather than silently mis-evaluated. Thrifty (Tiered Delegation for Claude Code) turns it into a cost discipline: a strong model writes the contract and sprint list, a cheap model builds and self-verifies against gates, and the strong model steps in only on failure — roughly 64% cheaper than running the strong model on the whole build, at equal quality. barnstormer builds living product specs as kanban cards with event-sourced, phase-gated workflows. Swamp Club productises the idea as typed workflows over versioned data. The extreme statement of the deterministic school is claude-ctrl: "an instruction that lives only in model context is not a constraint," so policy is enforced by runtime hooks that mechanically deny unsafe paths regardless of what the model remembers.

The two camps are not really enemies. The static side optimises for correctness, auditability and cost where the decomposition is known in advance; the dynamic side handles tasks where it is not. But the deterministic school currently holds the stronger empirical position, because it can point at measured cost reductions while the dynamic school mostly points at flexibility.

The economics are now measurable

Agent Swarm Model Economics gives the first detailed cost breakdown of coordination quality: Cursor's flat swarm cost $10,565 in API spend, while a frontier planner over cheap workers cost $1,339 — the entire worker fleet $411 — with workers carrying most of the tokens but planner tokens dominating cost. The SQLite-in-Rust experiment passed roughly two-thirds of sqllogictest in the first hour, and every model mix eventually reached 100%. The thesis: few moments in a large task genuinely require frontier intelligence, so split the work accordingly.

That tiering is now the consensus across independent deployments. Orchestrating AI Code Review at Scale routes the frontier model to the coordinator, mid-tier models to the workhorses and a cheaper model to text-heavy reviewers, landing a median review cost of $0.98. Thrifty (Tiered Delegation for Claude Code) is the same idea as a plugin. AI Code Migration with Claude Code is the same shape at migration scale: build a judge and a rulebook, then fan out translation to many agents, reserving the largest models for review. Uber — Agentic Engineering Shift reports costs up six-fold since 2024 and the identical mitigation. Fable Open-Sourced NanoClaw's PR Factory shows the flip side: five unattended hours and roughly $800 took a fork 405 commits stale eighty percent of the way to publishable — work that would have been weeks of engineer time.

Cost is only half the equation. AI Agents Need Clear Specs adds the specification dimension: in multi-agent pipelines underspecification is not sloppy, it is expensive, because interpretive freedom compounds at every agent boundary. The total cost of specification is U-shaped — minimal spec back-loads cost, full formal spec front-loads it — and the minimum sits around well-structured acceptance criteria. Building an Advanced Agentic Harness proposes the missing measurement instrument, multi-dimensional budgeting — tokens, tool calls, wall time and dollars tracked together, a single pressure() scalar driving graceful degradation — though it remains a design proposal rather than a deployed system.

Cross-harness orchestration and the control plane

A growing thread asks whether orchestration should span different agent harnesses, not just different models within one harness. bb — The Agent Orchestrator as Normalizer is the most principled answer: a two-shape adapter architecture that shells out to Codex, Claude Code, Pi and ACP agents and normalises each harness's traffic into a shared thread-event model, the runtime never interpreting provider-specific wire content. acpx provides one CLI surface over the Agent Client Protocol, with persistent sessions, prompt queueing and cooperative cancel. Warp Agent CLI delegates across Claude Code, Codex and others as interchangeable workers. Traycer wraps many agents under a versioned RPC protocol where backwards compatibility is a type-checked discipline. Omnigent makes the meta-harness argument in infrastructure terms: what Kubernetes did for servers, a meta-harness does for agents, making sessions, policies and skills portable across harnesses. The session managers — Orca, Agent of Empires, Dorothy, ctx – Agentic Development Environment — are the same idea one layer down, running many agent CLIs in parallel worktrees with a merge queue and a UI.

At the top of the stack the control-plane question becomes governance. Paperclip and OtoDock ship the company runtime: budgets with hard stops, approval gates, immutable audit trails — "autonomy is a privilege you grant, not a default." speedrift-ecosystem applies the same logic to a fleet of repositories — observe, prioritise, plan, execute, record — with bounded autonomy: it can restart safe services and emit corrective tasks under policy, but cannot rewrite unrelated local work without a trace. These are the few tools attacking the governance and cost-control gap rather than re-solving coordination patterns that already have answers.

Monitoring, triggers and the runtime beneath the agents

Two further layers sit outside the coordination taxonomy but are part of the orchestration problem.

Pi Observers introduces persistent observer agents that watch but never participate — triggered on lifecycle events, proposing short advisories, with a reconciler gating what reaches the main agent. Unlike workers in a hierarchy, observers never do the work; they are a quality-control layer running alongside it. AgentPulse is the operational complement: a drift-investigation tool that instruments turns and handoffs and flags when an agent's behaviour leaves its expected range across runs. The same adversarial instinct recurs everywhere a result must be trusted before it ships: TDD Coordinator (Corazonn)'s Rule of Two, Cloudflare Security Audit Skill's requirement that findings survive two independent refutation attempts, and Swarm Skill's three independent lenses, where a finding survives only if at least two do not refute it.

Beneath that sits the trigger and runtime question, which the field is only now taking seriously. Agents wait on humans, tools and webhooks, so an orchestration runtime must be able to pause a run, persist its state and resume without re-burning tokens. Agent Substrate pushes this to its limit with sub-second suspend/resume of agent "actors" multiplexed onto a small pool of sandboxed workers via gVisor and micro-VM checkpoint/restore — an early, deliberately ambitious answer to "where does the trigger land."

Where the Sources Disagree

Static decomposition versus runtime discovery. Tracker, Dippin (Language), maestro and Thrifty (Tiered Delegation for Claude Code) write the DAG up front, buying validation, simulation and cost measurement; Cord lets agents spawn and fork task trees at runtime, buying flexibility on tasks whose decomposition is not known in advance. Tracker's manager_loop handler only partly bridges the gap — the child structure stays static. The trade-off is genuine and task-dependent.

Centralised control plane versus per-repo graphs. speedrift-ecosystem argues for a central plane supervising work across repositories; workgraph argues for per-repo graphs where each repository keeps local authority. This mirrors the monorepo-versus-polyrepo debate and likely has the same answer: it depends on organisational structure.

Full automation versus visibility. Ralph, Gas Town's Agent Patterns and Paperclip push toward hands-off operation; Gas Town After 10,000 Hours of Claude Code argues delegation costs him visibility and agency, preferring pair programming, and Managing Agents via Kanban Boards keeps the human in the loop by design. This is a disagreement about the human's role, not the machine's capability.

Human approval versus bounded autonomy. weft requires approval for every state-mutating action; speedrift-ecosystem and Paperclip allow bounded autonomous action under policy with audit trails. The right answer depends on blast radius and reversibility, and no general framework yet decides which operation gets which treatment.

The orchestrator as product versus feature. Agent Orchestration for the Timid concludes, after testing five tools, that Claude-native skills and slash commands beat them all. Dorothy, Orca, Agent of Empires and klaw.sh are full products with real users. Loop Engineering offers the most satisfying resolution: practitioners find the patterns first, then the patterns become platform features — as when Claude Code shipped built-in task management and dependency tracking.

Cross-harness orchestration: bridge or permanent need. bb — The Agent Orchestrator as Normalizer, acpx and Traycer invest heavily in protocol normalisation. The open question is whether this is a temporary bridge until harnesses converge on common protocols, or permanent as harnesses differentiate rather than converge.

What's Missing

Cross-agent learning. Agents in orchestration systems do not learn from each other. If Worker A discovers a debugging pattern, Worker B starts fresh. Loomkin's Keeper processes, with failure memory and discoveries shared via PubSub, are the closest thing to shared learning, but they are built on Erlang/OTP (Process-Based Concurrency BEAM OTP) — the actor model, with isolated processes, message passing, supervision trees and per-process garbage collection, is what every agent framework ends up reinventing (Distributed Systems). The pattern has not been implemented in the mainstream Python/TypeScript ecosystem.

Coordination cost measurement. Agent Swarm Model Economics, Orchestrating AI Code Review at Scale and Thrifty (Tiered Delegation for Claude Code) offer isolated hard numbers, but there is still no shared instrument for comparing orchestration patterns. Building an Advanced Agentic Harness proposes a pressure() scalar that degrades gracefully across token, time and dollar budgets, but it remains a design, not a deployed system. Without measurement, choosing between orchestration patterns stays impressionistic.

Failure recovery at the orchestration level. Building an Advanced Agentic Harness contributes the cleanest error-classification primitive — classify_error() maps failures into transient, tool misuse, missing information and policy violation, and the recovery policy follows from the class. Loomkin's supervision trees are the most principled answer, but Python-based systems mostly retry or give up.

Multi-player orchestration. Every tool in the evidence is single-player. Zero Alignment identifies the gap — software is a team sport, but orchestration treats it as a solo activity with AI assistants. QM (Multiplayer Agent Harness), Munder Difflin — Clones of You, Not a Shared Bot and Moltbook are the first moves toward team-scale steering, but nothing is shipping that a team can use to coordinate agents and humans together.

Agent-to-agent security. Moltbook makes prompt injection the live hazard of agent networks — one agent's output is another agent's instruction — and OtoDock's "locked down by default, granted on purpose" posture is the exception, not the rule. Most orchestration tools do not sandbox the agents they run.

Also on This Theme

Compiled from 70 sources: bb Architecture Research Notes, How Grid/Patch Instructions Shaped Agent Research Depth, acpx — Headless CLI client for stateful Agent Client Protocol (ACP) sessions, A Practical Guide to Reducing Token Spend, Agent of Empires, Agent Orchestration for the Timid, Agent swarms and the new model economics, AgentPulse, How Anthropic runs large-scale code migrations with Claude Code, Multi-agent AI systems are organizations, Building an Advanced Agentic Harness, Building Reliable Agentic AI Systems, claude-ctrl, The guide to software factories, Cloudflare Security Audit Skill, TDD Coordinator (go.md), Cord, Show HN: ctx – an Agentic Development Environment (ADE), Dippin: A Domain-Specific Language for AI Agent Workflows, Dorothy, A Harness for Every Task: Dynamic Workflows in Claude Code, Event-Driven vs. Polling Architectures for Agent Triggers, Fable open sourced NanoClaw's agent factory. It cost $800., From one Claude agent to a fleet — in five small steps, fleet — Python supervisor for running coding agents in parallel, My thoughts on Gas Town after 10,000 hours of Claude Code, Gas Town's Agent Patterns, Design Bottlenecks, and Vibecoding at Scale, How Hightouch Built Their Long-Running Agent Harness, Introducing the Warp Agent CLI: a CLI coding agent that does what others can't, klaw.sh, Local Deep Research, Loomkin, Loop Engineering, maestro, Managing Agents via Kanban Boards, Mission Control: How We Built an AI Agent Squad with OpenClaw, Moltbook is the most interesting place on the internet right now, Munder Difflin — free, open-source multi-agent harness of personal clones, Introducing Omnigent: A Meta-Harness to Combine, Control and Share Your Agents, Orchestrating AI Code Review at Scale, Orchestrator - Worker Skill, OtoDock — the agentic company OS, Paca — AI-Native Open-Source Project Management, Paperclip, Paperclip — Open-Source Orchestration for Zero-Human Companies, The Complete Guide to Running Parallel AI Coding Agents, pi-observers, pi-subagents, poietic, Process-Based Concurrency: Why BEAM and OTP Keep Being Right, QM — a multiplayer agent harness for work, ralph-ban, Ralph (Ghuntley's Technique), Ruflo — Summary, Scaling Long-Running Agents, Sidekick Skill — Persistent Worker Agent for Claude Code, speedrift-ecosystem, Orca, Agent Substrate, Swamp Club, /swarm — Multi-Agent Swarm Orchestration Playbook, thrifty — A Tiered-Delegation Execution System for Claude Code, Tracker: Pipeline orchestration engine for multi-agent LLM workflows, Traycer (Summary), vibe-kanban, weft, What Ralph Wiggum Loops Are Missing (And When It Starts to Matter), Why AI Coding Agents Still Need Clear Specs, workgraph, Uber: Leading engineering through an agentic shift - The Pragmatic Summit Last compiled: 2026-09-12