Visa Vulnerability Agentic Harness (VVAH)

Visa's open-source 11-stage agentic SAST pipeline that discovers vulnerabilities, verifies them adversarially, proposes fixes, and validates those fixes — all driven by frontier LLMs. Built on Project Glasswing learnings, it targets Mean Time to Adapt (MTTA) as its primary metric, treating triage speed — not discovery — as the bottleneck in AI-assisted vulnerability management.

Architecture

VVAH is a linear stage pipeline with SQLite checkpointing, wrapped as a Python CLI (vvaharness). Eleven stages across four phases:

Phase 1 (S1–S3):  Attack surface mapping → threat modeling → hunt planning
Phase 2 (S4–S6):  Multi-lens deep-dive → deterministic pre-filter → adversarial verification
Phase 3 (S7–S9):  Dedup → exploit chaining → SARIF emission
Phase 4 (S10–S11): Remediation agent → adversarial validation panel

Each LLM-driven stage is a composable "skill" with its own system prompt and tool set. Stages are independently tunable and replaceable without rewiring the pipeline.

The core abstractions are: - ContextPackage (models.py:373): the structured understanding built by S1 — entry points, unsafe sinks, file-qualified call graph, deterministic file inventory — threaded through every subsequent stage - Finding (models.py:600): the central data contract, carrying chunk origin, file/line, vuln class, CWE, CVSS, confidence, and adversarial verdict - FinalReport (models.py:751): ranked findings + exploit chains + dropped-finding audit trail + scan metrics, rendered to Markdown and SARIF

The backend dispatcher (backends/llm.py:38) routes each pipeline role independently to via: cli (Claude CLI subprocess), via: sdk (Anthropic SDK), or via: openai (OpenAI-compatible). Every role in the pipeline can use a different backend and model. The default profile runs everything through Claude CLI — no API key needed, just a Claude Code login.

Key Techniques

Deterministic guardrails around LLM outputs

Every LLM-produced artifact is gated by deterministic code. The S1 agent surveys the repo, but the file inventory comes from a rglob() walk (s1_preprocess.py:158). The call graph is agent-emitted, then validated (hallucinated nodes dropped against source), then supplemented (regex-scans all source files for calls to known functions, building file-qualified edges like src/api/Controller.java::handleRequest → src/auth/AuthFilter.java::validateToken — often 3-5x more edges than the agent produced alone). This hybrid approach — LLM proposes, deterministic code verifies — runs through the entire pipeline.

Adversarial verification as the false-positive filter

S6 (s6_verify.py:17) spawns a fresh agentic Claude session for each finding and explicitly instructs it to prove the finding wrong. The verifier must: 1. Open the cited file at the cited line and establish what the code really does 2. Walk the call chain outward to an external entry point (no entry point → not exploitable) 3. Hunt for upstream protections (input validation, framework encoding, auth gates, feature flags) 4. Emit TRUE_POSITIVE/FALSE_POSITIVE + CVSS 3.1 vector + confidence (1–10)

This is more expensive per finding than confidence thresholds, but produces auditable verdicts with reasoning. The system prompt even specifies: "Confidence 8–10 means you actively searched for the opposite verdict and could not support it."

Temperature-gated majority voting

S4 deep-dive supports N independent runs per chunk with finding intersection, but _effective_runs() (s4_deepdive.py) checks whether the model actually supports temperature. Opus 4.7+/Sonnet 4.5+ reject explicit temperature — in that case N collapses to 1, preventing N× cost for identical output. To get real diversity-based voting, you must point models.deepdive at a temperature-capable model AND set temperature: 1.0. The shipped default profile has voting effectively disabled.

Config shape-hash dedup with secret safety net

S1's _dedup_configs() (s1_preprocess.py:370) collapses near-duplicate per-environment config files (e.g., 5,000 service/<svc>/<env>/config.yml variants) without pairwise comparison: 1. Hash each config's key structure (not values) — YAML indent-scan, JSON parse, INI sections, or generic KV regex 2. Cluster by shape hash, keep one representative per top-level dir 3. Safety net: re-read only the large clusters, diff each candidate's suspicious-value set against the representative's. If a variant contains a secret (API key, private key, JWT, Slack/GitHub token) or insecure value (TLS verification disabled, debug mode, anonymous auth) NOT present in the representative, it's promoted back into scope 4. Peak RAM is bounded to workers × max_file_kb, independent of candidate count

Crash-proof LLM output coercion

Every strict Literal/Enum/int field in the Pydantic models carries field_validator(mode="before") with extensive synonym maps (models.py:58-125). An off-schema LLM value like kind="rpc" maps to network, actor="attacker" maps to remote_unauth, confidence="high" maps to 0.9. These used to kill the entire run; now they degrade gracefully. The alias maps cover 30+ entry-point kinds, 30+ threat actors, 25+ sensitivity levels, and confidence values in 0-100%, 1-10 scale, or word form.

Design Decisions

Default edits source code. The shipped profile runs all 11 stages including S10 remediation in fix mode — vvaharness scan EDITS source files in the target repo. This is intentional: MTTA is the metric, and integrated fix generation closes the loop from discovery to validated fix in one run. Detection-only is --stop-after s9. The README carries prominent warnings.

Claude CLI as default backend, not SDK. No API key required — just a Claude Code login. The trade-off: no temperature control (majority voting disabled by default), reliance on an external binary. The upside is simpler onboarding, especially behind corporate gateways.

Checkpoint state lives outside the scanned repo (SQLite at ~/.vvaharness/state/). A hostile target could pre-plant state that --resume would load — not executable (JSON, never pickle) but pipeline-poisoning. Defense-in-depth keeps state outside.

Config from inside the target is refused. The canonical "cd into checkout, then scan" pattern would let an attacker-influenced config control pipeline behavior. VVAHARNESS_ALLOW_CWD_CONFIG=1 overrides.

Symlink containment is unconditional. S1 drops symlinks targeting outside the repo even when follow_symlinks is set — a committed symlink to ~/.ssh/id_rsa would otherwise exfiltrate host files into LLM prompts.

Validation (S11) is Anthropic-only. The adversarial panel uses Claude Agent SDK's deny-by-default permission sandbox. via: openai validate roles are refused.

Comparison Notes

Unlike deterministic SAST tools (Snyk, CodeQL) which find known patterns, VVAH uses LLMs to reason about data flow and exploitability, then adversarial verification to confirm. Unlike general coding agents (Claude Code Mastery, Oh My Pi (omp)), it's a specialized pipeline with fixed stage ordering and structured output contracts. It shares the "multi-agent with verification" pattern seen in Orchestrating AI Code Review at Scale but applies it to vulnerability discovery rather than code review.

The closest analogue in the wiki is Attack Review -- Claude Allowlisted-Egress Exfiltration, which also applies adversarial thinking to security — but VVAH automates the entire cycle at scale.

The "deterministic guardrails around LLM outputs" philosophy echoes the thesis of Guardrails and Feedback Loops ("linters beat prompts"), but applies it at the pipeline architecture level rather than the development workflow level.

Sources: raw/visa-vulnerability-agentic-harness Last updated: 2026-07-18