Raw capture — raw/swarm-skill.md

url: https://github.com/jleechanorg/claude-commands/blob/main/.claude/skills/swarm/SKILL.md title: "/swarm — Multi-Agent Swarm Orchestration Playbook" author: jleechanorg date_fetched: 2026-07-18 date_published: 2026-07

name: swarm description: Orchestrate multi-agent swarms (ultracode Workflow fan-outs + Agent-team lanes) with adversarial verification, /innov, /advice, and a sidekick durability layer for crash-recoverable runs — the playbook distilled from 2026-07 fable swarms

/swarm — Multi-Agent Swarm Orchestration Playbook

/swarm <goal> [--engine workflow|team] [--shape retro|review|solutions|innov|triage] [--sidekick [model]]

Run a goal as a swarm: a deterministic Workflow-tool fan-out (ultracode) or a named Agent-team of interactive lanes, with adversarial verification baked in.

The sidekick wrap is mandatory regardless of --engine — --engine only picks the fan-out mechanism the sidekick uses (Workflow tool vs agent-team lanes); it is never a way to skip the sidekick. --sidekick [model] only overrides the model (default: see Sidekick durability layer below).

Quick reference / defaults

Examples

# Monthly design retrospective swarm (Collect→Verify→Synthesize→Plan)
/swarm design retrospective for last month --shape retro

# Adversarial code-quality sweep of mvp_site (Review→Verify→Innovate→Docs)
/swarm code quality audit of mvp_site --shape review

# Innovation pass over existing design docs (Innovate→Challenge)
/swarm one smartest addition per design doc in docs/plans/<dir> --shape innov

# Crash-recoverable overnight fleet drive owned by a fable sidekick
/swarm fix CI health then drive all open PRs to green --sidekick fable

Domain-general by design. Nothing here is coding-specific: the same shapes (mine → adversarially verify → synthesize → gate) run market research, incident retros, literature reviews, ops audits, planning exercises, content pipelines. Substitute freely: "repo" → any durable shared destination (repo docs, ~/roadmap, wiki, gist); "PR" → whatever review surface the artifact ships to; "findings about code" → findings about anything with checkable evidence. The invariants that must survive substitution: refute-by-default verification against primary sources, disjoint writer outputs, commit-early durability, false-empty detection, a whole-artifact publishability gate, and cross-model cold review (rule 12). Distilled from 2026-07 design-retro swarms (5 workflows, ~180 agents, ~7M subagent tokens, worktree_fable_wa + worktree_fable_bulk).

Engine selection

"Ultracode" = the Workflow tool, never /team-claude teammates (user corrected this twice).

Stall watchdog — TUI swarms stall alive on quota modals (2026-07-11)

Interactive-TUI sidekicks AND every teammate lane stall SIMULTANEOUSLY on the per-account session-limit modal (/rate-limit-options), which never auto-dismisses after the quota window resets. Liveness+progress monitors miss it (process alive, no events). Supervisors MUST watch STATE.md staleness (~15 min alarm + pane-grep for session limit|rate-limit-options|Enter to confirm), and on recovery send Enter to every pane in the session (tmux list-panes), not just the lead — then have the lead re-task idle lanes via SendMessage. Full recipe: sidekick SKILL.md § "Stall watchdog (interactive TUI)". Cost of missing this: 3 lane-hours frozen, 2026-07-11.

Sidekick durability layer (Devin Fusion sidekick pattern)

DEFAULT (user directive 2026-07-11): the sidekick and its lanes run as named in-process teammates of the INVOKING session's Agent Team — visible in the user's panel, SendMessage-addressable ("I want it in this session"). External tmux sidekick sessions are the FALLBACK only (must-survive-session-exit missions / Agent Teams unavailable); durability in the default mode comes from STATE.md + a P1 resumption bead + commit-often, not process persistence. See sidekick SKILL.md § "DEFAULT MODE: in-session teammate".

ALL /swarm work runs inside the sidekick — always, no exceptions (user directive, 2026-07; reaffirmed 2026-07-10). The main session does NOT own the fan-outs — it spawns a sidekick (see ~/.claude/skills/sidekick/SKILL.md, command /sidekick) and the sidekick runs the swarm:

• Main session: writes/updates STATE.md, creates TaskList lanes, spawns the sidekick, relays milestone messages. Nothing load-bearing stays only in conversation context.
• Sidekick: reads STATE.md on start (never redoes logged steps), claims tasks, fans out cost-routed sub-agents per lane, appends to Progress Log + rewrites Next Actions after EVERY step, commits+pushes after every green unit (≤30 min uncommitted), and propagates the commit-often instruction verbatim into every sub-agent prompt.
• Branch-scoped STATE.md (mandatory): every sidekick/swarm mission gets its OWN state file at /tmp/<project-slug>/sidekick/<branch-name>/STATE.md (<project-slug> = repo name for repo missions, per the sidekick skill's derivation rules; /→- in branch names; mission slug for cross-branch missions). Parallel runs on different branches must never share a state file — sharing caused a 2026-07 clobber (retro sidekick overwrote the CI/fleet sidekick's live plan via the generic ## Next Actions heading). If a legacy shared /tmp/<project-slug>/sidekick/STATE.md exists: migrate only YOUR mission's sections to your branch-scoped path, leave a one-line pointer, treat everything else as read-only.
• 5-minute checkpoint cadence (MANDATORY for the sidekick AND every lane): at least once every ≤5 min, (1) append a timestamped heartbeat + current phase to STATE.md, (2) update the P1 mission bead body (br v0.2.16 has no native --append — read current notes via br show <bead-id> --json, concatenate the new heartbeat, write back with br update <bead-id> --notes "<combined text>") then br sync (br sync alone only syncs DB↔JSONL — it does NOT update the bead body on its own), and (3) make a local commit of state — so any crash loses ≤5 min. Drive it with a background timer/loop or CronCreate; a chat-only status print is NOT a checkpoint. Commit-safety: if the worktree carries unrelated staged/modified work (another actor's diff, a staged deletion), NEVER git commit/git add -A there — route to an isolated state repo (git init a .tmp/<mission>-state-repo/, copy state in, commit) or a WIP branch with path-scoped git add -- <state paths>; gitignored .tmp/ state needs the isolated-repo (or -f) path.
• Crash recovery: /sidekick [model] in any new session — one spawn, zero context re-derivation. Workflow-tool resumeFromRunId journals are same-session only; the sidekick's disk+git checkpoints are what survive.
• Durability boundary. The sidekick engine is a real Claude Code process in a real tmux new-session (see the sidekick skill's "Real primitive"), so it survives conversation crashes AND the parent CLI exiting — the tmux session runs independently of the orchestrating session's process tree. What it does NOT survive: host reboots, tmux kill-server, or /tmp cleanup. STATE.md + a tracking br bead (below) remain the actual durable state; the process is always respawnable from them via /sidekick.
• Model choice: sidekick itself may be fable (orchestration/diagnosis only, token-economy block required); its sub-agents are always sonnet/haiku. Engine choice: the sidekick may run on the codex engine (/sidekick codex ...) — same STATE.md/tmux protocol; codex lanes use its native multi_agent; between-run steering via codex exec resume <session-id> (see the sidekick skill's Engines section).

Quota-out handoff (MANDATORY) — any coding agent must be able to resume

Assume the session can hit provider quota at any moment. At mission start AND at every checkpoint, persist tasks/specs/goals/wave-plans in agent-agnostic form (no Claude-specific tool references; plain commands + file paths + PR URLs): 1. Primary: br beads — one P1 mission bead (current state, next actions, exact resume commands) updated at the same cadence as STATE.md, with a single-writer rule per mission bead (no concurrent-lane clobber). Any CLI (codex, cursor, gemini) can br search <mission> and resume. 2. Non-repo fallback path (when br unavailable): plain files under /tmp/<project-slug>/sidekick/<branch>/ (STATE.md + HANDOFF.md). 3. Durable artifacts (datasets, ledgers, specs) that another agent needs: commit + push them to the working branch — /tmp dies with the host, git does not. A resume must be possible from the bead body ALONE: repo/branch, task list with statuses, wave plan, blocking items, and the literal next 3 commands to run.

Bead runbook — cross-session recovery beyond /tmp

STATE.md under /tmp/<project-slug>/sidekick/<branch>/ does not survive a host reboot or /tmp cleanup. At mission start, create (or update) one P1 br bead as the durable, host-independent recovery pointer: session id + repo/branch, script scratchpad paths, live workflow runIds, output-dir + doc-numbering state ("docs go to docs/plans/<retro>/pr-retro/ numbered from 05"), and pointers to STATE.md + any memory files. A completely fresh session with no /tmp file left can br search <mission-keyword> and reconstruct full context from the bead body alone. Real example: rev-ewnuu ("Design-retro 2026-06 swarm completion...") — written 2026-07-07, contains exact resume steps for both the pr-retro and code-quality lanes plus the close-out checklist. Update the bead at the same cadence as STATE.md; close it only when the mission's close-out step (province comment + /learn) actually lands.

Canonical phase shapes (all proven 2026-07)

Hard rules (each one cost a failure to learn)

1. Cost-route every agent() call explicitly — verify with a grep, not a memory check. Workflow agents inherit the session model — an unrouted 40-agent fan-out on a Fable session runs entirely on Fable. model: 'haiku' for mechanical merge/cluster/ls stages, 'sonnet' for miners/verifiers/doc writers; main loop (Fable) only for final synthesis. Before launching any script, run grep -n "agent(" <script> and confirm every match either has an explicit model: in its opts or is the deliberate final-synthesis call — an eyeballed "looks routed" pass misses calls added late or inside a loop/helper. Real incident (2026-07-07): a token-usage audit on jeff-ubuntu found one session mixing 428 MiniMax-M3 messages with 367 claude-fable-5 messages — an unrouted fan-out had silently run its mechanical mining/verify stages on the parent session's premium model for weeks before anyone noticed, because nothing checked the script for missing model: before launch. If a stage's cost-routing is genuinely uncertain (e.g. a helper function wraps agent() and the model arg is threaded through indirectly), do NOT assume it's fine — print/log the resolved model for the first agent in each stage at runtime and confirm it in the phase output before trusting the rest of the fan-out.
2. Adversarial verify, refute-by-default. Every candidate finding faces ≥3 independent lenses (e.g. evidence / design / severity) prompted to REFUTE; survive only on ≥2/3 non-refuted. Dedup vs a seen set, not vs confirmed.
3. Audit verifier-death false kills — and detect false-empty completions. A verifier that died (rate limit, API error) is NOT a refutation. After any run with agent failures, re-verify findings whose "refutations" were dead agents before accepting the kill. This generalizes to whole-workflow results: a run that returns 0 confirmed findings is a VOID, not a verdict, if its failures[] list shows mass agent death (e.g. every rejected[].why reads DEAD_VERIFIER/verifier died, or the failures array is dominated by Server is temporarily limiting requests). Distinguish the two before reporting: 0 findings + all agents healthy = real "nothing here" result; 0 findings + mass 429/error failures = the mining/verify stage never actually ran — re-run it, don't ship or record the empty result. Hit twice on 2026-07-07: pr-retro-gapfill (15/15 verify agents died, 5 real Collect-stage findings falsely zeroed out) and code-quality (60/60 verify agents died, 20 real Collect-stage findings falsely zeroed out) — both had genuine candidate findings from a healthy Collect phase; only Verify was VOID.
4. Rate-limit resilience — concurrency is aggregate, not per-workflow. Provider 429s ("Server is temporarily limiting requests") kill whole fan-outs, and the trigger is total agents in flight across ALL concurrently-running swarms on the account, not any single workflow's size (confirmed 2026-07-07: two separate workflows, each individually reasonable, hit 429 in the same window once their combined in-flight count reached ~75). Rules: (a) do not launch a new multi-agent fan-out while another large workflow is mid-fan-out — serialize big stages across sibling swarms, not just within one; (b) on mass 429, stop, start a ~20-min background cool-down timer (sleep 1200 run_in_background or ScheduleWakeup), then retry via a sibling-workflow-as-canary: a tiny throwaway 1-agent workflow (not just "the smallest of the affected workflows") used purely as a health probe, cheaper than blind-resuming the real fan-out; only relaunch/resume the real workflow(s) after the canary succeeds; (c) while cooling down, keep any other concurrently-running lane to ≤4 concurrent sub-agents; (d) stagger large fan-out starts — bursts kill, ramps don't. Root-cause analysis of three separate 429 storms in one evening showed every kill coincided with an INSTANT N-wide parallel() fan-out (16-, 60-, and 15-wide simultaneous cold starts), while gradual ramps (pipeline(), which naturally staggers stage-by-stage) and ordinary serial work were never throttled. Inside a large fan-out, chunk the thunk array into batches of ~8 with a few seconds between batches instead of one parallel() call over the whole array; treat ~16 simultaneous cold starts as roughly the survived ceiling, not a safe target to build up to. Never start two different workflows' fan-out stages within the same minute, even if each individually looks small.
5. Resume semantics. Persist scripts to the session scratchpad (cp from the returned Script file path) so they survive crashes. To resume: TaskStop the run, edit only NOT-yet-completed agent() calls (touching a completed call's prompt/opts invalidates its cache), relaunch with {scriptPath, resumeFromRunId}. Resume journals are same-session only — after a crash, re-run fresh from the persisted script. Cross-session / cross-reboot recovery needs more than the script: write a P1 br bead (see "Bead runbook" above) with session id, script paths, live runIds, and output-dir/doc-numbering state, since /tmp and same-session journals both vanish on a host reboot or /tmp cleanup.
6. Disjoint output dirs per lane. Each workflow writes to its own OUTDIR (docs/plans/<retro>/pr-retro/, .../code-quality/, .../innovations/); parallel workflows must never share a mutable file. Each doc-writer agent may write EXACTLY ONE file.
7. Commit + push artifacts as they materialize — two layers. (a) Each doc-writer agent commits its own single file immediately after writing it (git add <path> && git commit), so partial progress survives even if the workflow itself dies mid-run before ever returning. (b) The workflow's final phase includes one explicit Commit agent that sweeps any stragglers and pushes (git status --short for anything the per-doc commits missed, then git push) and reports the resulting HEAD SHA — never rely on "the doc writers probably committed everything" without a final agent that verifies and pushes. Never leave swarm output only in /tmp or the session dir; repo docs (docs/plans/...) or ~/roadmap/ are the durable homes.
8. Pre-build datasets in the main loop. Miners read a static dataset JSON (e.g. prs-last-month.json built once via gh) instead of each making live API calls — deterministic, cheap, rate-limit-proof.
9. Heartbeats + time-box, cadence scaled to what you're waiting for. Don't default every wakeup to ~270s — that's for tight external polling loops (CI runs, deploys) where the cache stays warm and the state changes fast. For a long multi-agent fan-out (tens of agents, many minutes), poll on a 1200-1800s cadence instead; checking every 270s on a 10-minute workflow just burns cache misses for no new information. Obey the 3-hour autonomy time-box (user-scope rule) regardless of cadence.
10. Finish with /advice. Final package gets an /advice pass (3 parallel reviewers: Opus subagent, /research, /secondo) before the closing PR comment; then /learn.
11. Publishability gate — mandatory final stage, after ALL writer lanes, before the PR is called done. Adversarial verification up to this point attacks candidate findings, not the rendered docs that actually ship — a healthy 3-lens verify pipeline can still publish a doc with a leaked credential path, a stale metric, a forbidden recommendation, or a contradicting sibling doc, because no lane ever re-reads the finished markdown as a whole. Run one final single-agent (or small dedicated) gate over the ENTIRE docset that checks: (a) redaction sweep — rg 'file:///Users|/Users/[a-z]+|ghp_|gho_|x-access-token|serviceAccountKey' over every published doc, replace machine paths/token forms with generic placeholders; (b) cross-doc consistency — every numeric claim in a top-level/summary doc is diffed against its per-finding source doc, disagreements block; (c) freshness re-baseline — every present-tense claim re-checked against the PR's current head SHA, historical evidence explicitly marked "at base <sha>"; (d) supersession markers — each finding has exactly one authoritative doc, earlier superseded drafts carry a banner pointing to it; (e) policy lens — recommendations checked against repo-specific hard law (e.g. ZFC / ZFC-leveling file boundaries, credential-discipline, regex-approval bans) that the evidence/severity/design lenses never covered; (f) recipe validity — every copyable command/acceptance-check states the CORRECT expected outcome (e.g. a negative test should assert red, not green); (g) mechanical hygiene — git diff --check clean. Real incident: a cold review of the 2026-07-06 design-retro docset (~180 agents, 5 workflows, 7/10+ findings confirmed) found all six of these classes present in already-"confirmed" docs — see docs/design-retro-2026-06-adversarial-gaps.md for the full writeup and the six follow-up beads. The gate is planning-scope (a workflow stage + checklist), not production code.

12. Cross-model cold review — MANDATORY, not "when available," for any code-behavior or merge-readiness claim. A same-model swarm shares one set of blind spots: ~180 Claude agents (2026-07-06) unanimously passed a docset in which a codex cold review then found 6 real defect classes, because every swarm agent inherited the same priors about what "verified" means. This recurred on 2026-07-10: a same-model /er + /advice pass (both Claude) signed off on a "fixed" production bug; a single codex adversarial pass immediately found 5 real blocking defects the same-model pass missed entirely — data corruption in an unrelated field, an incomplete guard, meaningless timestamp semantics, unproven causation, and a broken existing test contract — because codex executed the code and reproduced the bugs rather than reading the diff and pattern-matching against the PR's own framing. Before any "ready to merge" / "bug is fixed" / "evidence PASS" claim is reported (not just after a publishability gate on a docset), dispatch at least one DIFFERENT model family as a genuinely adversarial reviewer — codex CLI (codex exec --dangerously-bypass-approvals-and-sandbox), gemini CLI, cursor-agent, whatever is installed (/advice and /secondo already provide multi-model routes for lighter-weight decisions, but a merge-readiness claim on real code needs the full adversarial pass, not the 3-reviewer /advice summary alone). Prompt it explicitly to (a) try to reproduce each defect class by executing code, not just reading it, (b) check for scope creep beyond the PR's stated claim, (c) check whether the evidence proves causation or only correlation. Treat its findings like any candidate: independently reproduce the most concrete/checkable one yourself before accepting the rest at face value (a single spot-check that confirms is strong evidence for trusting the whole report; a spot-check that fails means re-verify everything). Never skip the pass because "the swarm already reviewed it" or "we're in a hurry" — model diversity catches what intra-model redundancy structurally cannot, and skipping it is exactly how a broken fix ships.

13. CI-triggering fan-outs have a SEPARATE capacity ceiling from Claude/provider API rate limits — self-hosted runner/host capacity, not just agent concurrency. Rule 4 covers Claude/Anthropic API 429s from agent concurrency; this is a different resource entirely. Any swarm whose subagents call gh pr update-branch, push commits, or otherwise trigger fresh CI runs can overwhelm a fixed-size self-hosted runner fleet even when the agent fan-out itself is small and well-throttled (each "dumb" agent is one cheap API call, but each successful action can trigger 5-30+ downstream CI jobs). Real incident (2026-07-07, your-project.com): a 68-item gh pr update-branch sweep (one haiku agent per PR, ~1 minute wall-clock, zero Claude-API rate-limit issues) triggered 26+ simultaneous CI jobs against a 16-runner host and crashed the host outright (hard reboot, uptime reset to ~1min) — the fan-out itself was healthy, the DOWNSTREAM system it triggered work on was not sized for the burst. Before launching any swarm whose action multiplies into external CI/build/deploy triggers, check the target system's current capacity/queue depth first, and batch the trigger in chunks (e.g. ~15-20 at a time with pauses) rather than firing all N at once, even if N is small enough to be a trivial Claude-API fan-out on its own.

14. Max-parallelization sweep — a blocked gate blocks only its dependents, never sibling work (2026-07-13). Under-utilization is a failure mode equal to collision. Three sub-rules, each learned from the DK2D r2 session where 5 disjoint evidence-prep items sat idle for ~an hour while everything waited on one critical-path fix: (a) Standing re-scan: at EVERY coordination touchpoint (lane report, steer, heartbeat, user ping) re-scan the backlog (br ready, open beads, triage tables, STATE next-actions) for dispatchable work that is file-disjoint from active writers — independence is checked continuously, not just at spawn time. (b) Gate scope discipline: when a verdict blocks an artifact (e.g. "evidence run blocked until fix + PASS"), enumerate exactly which items depend on the gated artifact; everything else stays dispatchable — never let a gate expand into a planning-horizon freeze. Prep work that the gated artifact will CONSUME (harness hardening, ledgers, design docs, citations) is the highest-value parallel work because it converts gate-lift into instant execution. (c) Utilization metric in STATE: log lanes-active / items-dispatchable at each touchpoint; if dispatchable > active for two consecutive touchpoints, spawn lanes or launch an ultracode Workflow fan-out (bounded one-shot tasks → Workflow with explicit model: routing; steerable/long-lived → teammate lanes). Foreign/concurrent writers narrow the DISJOINT set — they never justify dispatching nothing.

Execution recipe

1. Instant start (mandatory UX): before ANY analysis — write the mission STATE.md from the goal text, then spawn the worker lanes as OFFICIAL Agent Teams teammates IN THE CURRENT SESSION (Agent tool + name: lane-<n>-<topic>; visible in the user's own team UI, SendMessage-addressable, registered in ~/.claude/teams/session-*/config.json — the real /team-claude-class feature), and spawn the tmux sidekick as the durability keeper (STATE.md owner + crash respawner). The user SEES the team in their own window within the first minute — a team hidden inside the sidekick's tmux session does NOT satisfy this. Headless/unattended runs invert: lanes live inside an interactive-TUI sidekick instead.
2. Recall context first: /history <goal keywords> + /ms <goal keywords> — find prior swarms, datasets, and findings before spawning anything else. Session JSONLs may live under ~/.claude-wa/projects/ (not only ~/.claude/projects/); workflow journals under <project>/<session-id>/workflows/*.json. 1.5. Sidekick wrap (mandatory, no exceptions): write/refresh STATE.md, spawn (or reuse) the sidekick via /sidekick (real tmux Claude Code process; steered via STATE.md + tmux, never SendMessage — it's a separate session). Division of labor: in an ATTENDED session the visible in-session Agent Teams lanes (step 0) do the work and the sidekick keeps durable state + respawns after a crash; in UNATTENDED/headless runs the sidekick owns ALL lanes itself (interactive-TUI sidekicks run them as a real Claude team; -p sidekicks fall back to Agent-tool subagents). STATE.md path resolution for swarm missions: on a git feature branch, use <branch> per the sidekick convention; on main/no branch, derive the mission slug with the sidekick skill's OWN rule applied to the full goal text (first 4–6 significant words, lowercased, kebab-case, ~40 char cap) — never a --shape-specific variant, so a later bare /sidekick crash-recovery invocation recomputes the SAME slug. Before spawning, ALWAYS ls the computed STATE_DIR: existing STATE.md → reuse/respawn the existing sidekick; missing → fresh spawn. Never let two derivations of the same goal spawn two concurrent sidekicks owning one OUTDIR.
3. Scope lanes and check independence (disjoint files/OUTDIRs). 2.5. Continuous utilization check (rule 14): at every subsequent touchpoint, re-scan for idle dispatchable work; blocked gates block only their dependents; log lanes-active/items-dispatchable in STATE and fan out when under-utilized two touchpoints running.
4. Author the Workflow script(s): meta with named phases, schemas for structured outputs, explicit models, single-file write permission per doc agent. 3.5. Cost-route gate (mandatory, run before launch): grep -n "agent(" <script> — every match needs an explicit model: unless it's the deliberate final-synthesis call on the main-loop model. Fix any bare agent() call before proceeding; do not launch on a "looks fine" read of the script.
5. Launch in parallel (independent lanes in one message), copy scripts to scratchpad, record runIds in tasks (TaskCreate per lane).
6. On completion per lane: commit+push artifacts, audit false kills, mark task complete. 5.5. Cross-model adversarial pass (mandatory, rule 12): for any lane whose deliverable is a merge-readiness/bug-fixed/evidence-PASS claim on real code, dispatch a genuinely different model (codex/gemini/cursor-agent) to adversarially attack the claim — reproduce, don't just read. Spot-check the most concrete finding yourself before accepting the rest.
7. Close: /advice pass → PR comment with provenance (runIds, agent counts, token spend) → /learn.