Building a Production AI PR Review Agent

A GitHub gist holding two things: a set of structured notes (key points, quotes, tools/practices, unanswered questions) and the full auto-generated transcript of a video course on designing and building a production AI pull-request review agent. The transcript is unedited ASR — names are garbled, sentences restart, and "idempotency" appears as "item potency" and "unaltered key" throughout.

The course splits cleanly in half. The first half is system design done out loud, from first principles, before any code. The second half is a screen-recorded build session driven by the speaker's own agent harness, "Genesis Kit."

The thesis

The problem is not automating code review. It is selectivity: which findings the agent takes, and which ones a scarce human judgment gets spent on.

The framing device is to delete the component and feel the pain. Remove the AI reviewer and what remains is a senior engineer with 20 PRs to get through, where "the 10th review of the day is not the first" — fatigue, inconsistency, and PRs queued for days. So the system exists to reclaim senior reviewer attention by automating the mechanical part, not to comment on every line. "Selectivity is the seed of this whole architecture."

The design loop, run once per component

1. Map the mess — document what a human reviewer does today, every micro-decision, where they recall knowledge, where they fail.
2. Name a precise trigger and a structured output — not "a claim comes in" but "an email to this address containing the word claims." Here: the GitHub pull_request webhook in, and a structured review out.
3. Assign each step a component — webhook/API, LLM, deterministic ML, retrieval, human checkpoint.
4. Choose an autonomy level — a golden rule that money, legal, or health forbids full autonomy; start with more human involvement than you think you need and withdraw it as the system matures.
5. Ask what can go wrong, in two directions — engineering (GitHub retries, forged payloads, 10-second ack budget vs. 90-second agent) and LLM (hallucination, wrong retrieval slices, tool timeouts).

The failure-mode catalogue is the strongest part: hallucination, model drift, tool/API timeout, feedback-loop poisoning (a junior's bad review teaches the agent), orchestration deadlock (aggregator waits forever on a hung specialist), human bottleneck (queue grows faster than reviewers clear it), and the "almost right" problem — 90% correct, 10% subtly wrong.

The architecture

GitHub webhook → FastAPI ingress (HMAC signature verification, idempotency-key dedup, immediate ack) → Redis + ARQ queue → LangGraph orchestrator fanning out via the Send API to four specialists (security, quality, testing, docs) running in parallel, each grounded by retrieval → aggregator that merges, deduplicates, and scores → confidence gate → auto-post to GitHub above threshold, human approval queue below it or on any critical finding.

The unit that travels between components is a finding: agent_type, severity/category, file/line, confidence, rationale. Confidence drives the gate; rationale makes the finding auditable and disputable when the confidence itself turns out to be wrong.

The four specialists are derived, not chosen — they come from the four things a human reviewer does: bring codebase context (→ retrieval), reason across separate concerns (→ multi-agent), stay skeptical, and cite evidence (→ every finding carries rationale and confidence).

The data layer

Three memory shapes, from the human analogy: semantic (the codebase itself), episodic (past reviews, what was flagged and disputed), procedural (team conventions). Observability adds a fourth: time-series spans for every LLM call, tool call, cost, and latency.

The argument is that three databases means three sets of connection pools, backups, failure modes, and corruption risks — and a single question ("for this PR, what did we retrieve, what did we produce, what did it cost?") requires stitching three systems together in Python. The answer chosen is one managed Postgres (TigerCloud) carrying all four shapes: pgvector for embeddings, pgvectorscale/DiskANN for index-on-SSD at millions of chunks, TimescaleDB hypertables for the append-only event spine, and continuous aggregates so the cost dashboard reads a summary table instead of scanning 10 million rows. TigerCloud is also the course sponsor, with a credits link.

To avoid framework lock-in, LangGraph sits behind an abstract WorkflowEngine interface (run / resume / get_state) so Temporal can be swapped in later. LangGraph was chosen over Temporal for the MVP because it runs in-process, has first-class fan-out, and checkpoints to the Redis already present for the queue.

Genesis Kit

The build half runs on the speaker's own harness. Its pieces: a locked DONE.html spec no agent may touch (because agents drift the goal to suit themselves); a context graph of invariants (every webhook passes HMAC; every finding carries confidence and rationale; the events table is append-only); implementation-notes.html as live state so a fresh session knows what is already built; a plan.md of milestones each with a demo command, success criteria, loops, skills, and a token budget; five gates (G0 through L4) with build, debug, and research loops; and an independent verifier agent spawned with fresh context that gets only the goal, success criteria, and invariants — never the builder's trail — on the reasoning that letting a model write and mark its own exam paper produces only the questions it already knows.

There is also a "quiz me" protocol that interrogates the human about the code before a milestone is accepted. It earns its keep on camera: the verifier catches a malformed-but-validly-signed JSON payload crashing with a 500 instead of a 400, and a TRUNCATE path that bypasses the row-level delete trigger protecting the append-only events table. The quiz then surfaces a live configuration bug — the schema declares vector(1536) while .env sets an embedding dimension of 256, a mismatch that would fail at insert time several milestones later.

The speaker is explicit that this is not vibe coding, and also explicit about token economy: use Opus as an orchestrator and spawn Haiku or Sonnet for mechanical work. "If you have generated one token, tell me what was the output of that."

What the source leaves open

By its own admission and by omission: how conflicting specialist findings are resolved (the aggregator "merges and deduplicates," but no conflict logic is given); chunking, embedding-model choice, and hybrid-search tuning; capacity planning for the human approval queue; how the golden dataset and regression gate are built and maintained; concrete drift detection; prompt injection from adversarial PR diffs (named as a trust boundary, not explored); token economics beyond an aggregate budget check; and multi-repository scale — the architecture assumes one repo. Deployment is named (Railway) but secrets management, alerting, and disaster recovery are not discussed.