Raw capture — raw/visa-vulnerability-agentic-harness.md
url: https://github.com/visa/visa-vulnerability-agentic-harness title: Visa Vulnerability Agentic Harness (VVAH) author: Visa, Inc. date_fetched: 2026-07-18 date_published: 2026
VVAH (vvaharness) is Visa's open-source harness for autonomous vulnerability discovery, remediation, and validation using frontier AI models. Built on learnings from Project Glasswing (Anthropic's initiative for AI-assisted vulnerability research), it runs as a four-phase, eleven-stage pipeline from code ingest to validated fix. It is a Python package (≥3.10) with multi-model support via vendor-neutral backend abstraction, outputting Markdown + SARIF 2.1.0 reports.
Version: 1.1.0, Apache 2.0 license.
vvaharness/
__init__.py, __main__.py, cli.py, models.py, manifest.py, agentdoc.py
config/ — YAML config loader with ${ENV} expansion, profiles (default/sdk/full)
orchestrator/ — pipeline driver: entry.py, scan.py, batch.py, checkpoints.py, store.py, cleanup.py, ...
pipeline/stages/ — s1_preprocess, s1_autoexclude, s2_threatmodel, s3_decompose,
s4_deepdive, s5_prefilter, s6_verify, s7_dedup, s8_chain, s11_validate
backends/ — llm.py (dispatcher), sdk.py (Anthropic SDK), agent_sdk.py (Claude Agent SDK),
oai.py (OpenAI-compatible), claude_cli.py (Claude CLI subprocess),
localtools.py (sandboxed tool loop)
remediation_agent/ — Step 10: runner.py, remediate.py, playbook.py, prompts.py, ...
validation/ — Step 11: Claude Agent SDK adversarial panel
report/ — enrich.py, cvss.py, cwe.py, redact.py
injectors/ — cve_feed.py, design_controls.py
util/ — environment, tokens, metrics, errlog, prompts, json_extract, status
lang/ — language hints for ~20 languages
The pipeline is a linear stage pipeline with SQLite checkpointing, allowing resume from any stage:
Phase 1 — Discovery & Modeling (S1–S3) - S1: Agentic exploration via Claude CLI with Read/Glob/Grep/Bash tools → ContextPackage. Includes deterministic file walk, config-deduplication, and call-graph supplementation. - S2: Threat modeling (STRIDE, OWASP, trust boundaries) → ThreatModel - S3: Decomposition into prioritized chunks (risk/taint/catch-all/specialist) → TaskManifest
Phase 2 — Deep Dive & Verification (S4–S6) - S4: Per-chunk deep-dive with majority voting (when temperature >0). Each chunk researched N times, findings intersected. - S5: Deterministic pre-filter + optional semantic pre-dedup - S6: Adversarial verification — fresh agentic session for each finding, tasked with PROVING IT WRONG. Emits TRUE_POSITIVE/FALSE_POSITIVE + CVSS 3.1 vector.
Phase 3 — Synthesis & Reporting (S7–S9) - S7: Two-pass dedup: deterministic (same file + vuln_class + line proximity) then semantic (LLM collapse of same-root-cause findings) - S8: Exploit-chain construction + severity ranking → FinalReport - S9: Markdown → SARIF 2.1.0 conversion (fully deterministic)
Phase 4 — Remediation & Validation (S10–S11) - S10: Remediation Agent — per-finding fix proposal/application with playbook-per-CWE-language-framework - S11: Agentic adversarial validation panel (security-architect + penetration-tester + optional cross-repo-analyzer)
1. Deterministic + LLM hybrid for reliability
Every LLM output is gated by deterministic checks. S1 surveys the repo agentically, but the file inventory comes from a deterministic rglob() walk that cannot be fooled. The call graph is agent-emitted, then validated (hallucinated nodes dropped), then supplemented (regex-scans source for known function calls). S5 pre-filter applies deterministic confidence/evidence gates before the expensive adversarial verification.
The philosophy: the LLM proposes, the pipeline verifies. This is most explicit in s6 adversary verification — a fresh agent is told "assume the finding is WRONG until you personally confirm it" and must trace call chains to an external entry point.
2. Multi-backend abstraction with role-level routing
The backend dispatcher (backends/llm.py) routes each pipeline role independently: cli (Claude CLI subprocess), sdk (Anthropic Python SDK), or openai (OpenAI-compatible API). Each role can use a different backend and model. The default profile uses via: cli for everything (reusing Claude Code login), while sdk.yaml routes everything through the Anthropic SDK with temperature-based majority voting enabled.
The config model:
models:
preprocess: {id: "claude-sonnet-4-6", via: cli}
deepdive: {id: "claude-opus-4-6", via: sdk, temperature: 1.0}
verify: {id: "claude-sonnet-4-6", via: cli}
remediate: {id: "claude-opus-4-6", via: cli}
3. Temperature-gated majority voting
S4 deep-dive runs each chunk N times and intersects findings. But _effective_runs() checks whether the model actually supports temperature — if _supports_temperature() returns False (Opus 4.7+/Sonnet 4.5+ reject explicit temperature), N collapses to 1 even if config says 3 runs. This prevents paying N× cost for identical output. The designer must choose a temperature-capable model AND set temperature: 1.0 to get real diversity-based voting.
4. Config dedup via shape hashing
S1's config-deduplication is a clever optimization for repos with many near-identical per-environment configs (e.g., 5000 service///config.yml variants). It: - Hashes each config's key structure (not values) using format-specific parsers (YAML indent scan, JSON parse, INI sections) - Clusters files by shape hash - Keeps one representative per top-level dir, plus any variant whose body contains a secret or insecure-value pattern NOT present in the representative - Uses a secret regex safety net (API keys, tokens, private keys, JWT patterns, Slack tokens, GitHub tokens) and insecure-value detection (TLS verification disabled, debug mode, anonymous auth)
5. File-qualified call graph supplement
The agent-emitted call graph is bare function names. S1 supplements it by:
- Scanning every in-scope source file for function definitions (8 regex patterns across ~6 languages)
- Building a def-site index (function → set of files with line numbers)
- For each caller-callee edge: qualifies bare names with file paths, resolving ambiguous callees by longest-common-directory-prefix
- Iterating regex-based call discovery for N rounds to expand the graph
- Output is qualified edges: src/api/Controller.java::handleRequest → src/auth/AuthFilter.java::validateToken
This turns a sparse, unvalidated LLM graph into a dense, file-qualified, source-verified graph with ~3-5x more edges than the agent produced alone.
6. Crash-proof LLM output coercion
Every field that takes strict Literal/Enum/int types from LLM JSON carries field_validator(mode="before") that maps common synonyms to valid members and falls back to a safe default. Off-schema values (e.g., kind="rpc", size="tiny", confidence="high") used to kill the entire run. Now they degrade gracefully. The coercion maps are extensive: _EP_KIND_ALIAS maps 30+ strings to 6 entry-point kinds; _ACTOR_ALIAS maps 30+ roles to 7 threat actors; confidence values accept 0-100%, 1-10 scale, NaN, and clamp to [0,1].
1. Default edits source code (fix mode)
The shipped default profile runs all 11 stages including S10 remediation in fix mode — meaning vvaharness scan EDITS source files in the target repo by default. This is an unusual choice for a security tool. The README carries prominent warnings, and --stop-after s9 provides a detection-only path. The rationale: the primary metric is Mean Time to Adapt (MTTA), and the bottleneck is triage speed, not discovery. By integrating fix generation into the pipeline, findings go from discovery to validated fix in one run.
2. Claude CLI as default backend (not SDK)
Despite shipping a full Anthropic SDK backend, the default profile uses the Claude CLI subprocess (via: cli). This means no API key is required — just a Claude Code login. The trade-off: no temperature control (so majority voting is disabled by default), and reliance on an external binary. The upside: simpler onboarding, works behind corporate gateways that already support Claude Code.
3. Checkpoint state outside the scanned repo
Checkpoints live in $VVAHARNESS_STATE_DIR/vvaharness.db (SQLite), deliberately outside the scanned repo. The rationale is defense-in-depth: a hostile target could pre-plant checkpoint state that --resume would load. Payloads are JSON (never pickle), so planted bytes can't execute code — but they could poison the pipeline, so state stays outside.
4. Symlink containment is unconditional
S1's file walk drops symlinks whose target resolves outside the repo unconditionally — even when follow_symlinks is explicitly set. The rationale: a committed symlink to ~/.ssh/id_rsa or /etc/passwd would otherwise pull host file content into the LLM prompt. No config flag can re-open this guard.
5. Config from inside the target is refused
By default, VVAH refuses to load config.yaml if it resolves inside the scan target directory (canonical "cd into the checkout, then scan" pattern). This prevents an attacker-influenced checkout from controlling pipeline behavior. The operator must set VVAHARNESS_ALLOW_CWD_CONFIG=1 to override.
6. Validation is Anthropic-only
S11 validation uses the Claude Agent SDK exclusively — via: openai validate roles are refused with exit code 2. Similarly, remediation fix mode (which needs Edit/Write tools) is effectively Anthropic-only since the OpenAI backend is sandboxed to Read/Glob/Grep. This is a practical constraint, not an architectural one — the Claude Agent SDK's permission sandbox (deny-by-default, no Bash, explicit tool grants) aligns with the validation use case.
7. Single-pass by default (no majority voting)
The shipped default uses via: cli with no temperature control, so S4 runs each chunk once. Majority voting requires an SDK backend + temperature-capable model. The trade-off: single-pass is ~3x cheaper per chunk, and s5+s6 provide the false-positive defense instead.
Hybrid deterministic/LLM pipeline: The deterministic guardrails (file inventory, call graph validation, config dedup, symlink containment) aren't afterthoughts — they're deeply integrated, with the LLM proposing and deterministic code verifying. This is unusual in agentic tooling, where people typically trust the LLM or don't use one at all.
Adversarial verification as the FP filter: Rather than fine-tuning or prompt engineering for accuracy, VVAH spawns a fresh agentic session for each finding and explicitly instructs it to "prove the finding wrong." The adversary must trace call chains to external entry points, hunt for upstream protections, and emit a CVSS vector. This is more expensive per finding but produces auditable verdicts with reasoning.
Config shape-hash dedup with secret safety net: The two-pass approach — first hash the key structure, then re-read only large clusters to diff suspicious values against the representative — is efficient (no O(N²) pairwise diff) and safety-conscious (no secret-bearing file is silently dropped).
File-qualified call graphs from regex + heuristics: Most SAST tools either use static analysis (precise but language-specific) or LLM reasoning (language-agnostic but imprecise). VVAH's hybrid approach — start with LLM guesses, validate against source, then supplement with regex — gets good-enough call graphs for ~20 languages without a compiler.
Relative to other agentic coding tools (like those documented in the wiki's Agent Design section), VVAH is distinctive in several ways:
vs. CI/CD security scanners (Snyk, CodeQL): These are deterministic but shallow — they find known patterns, not novel exploit chains. VVAH uses LLMs to reason about data flow and exploitability, then adversarial verification to confirm.
vs. general coding agents (Claude Code, Codex, Copilot): These are general-purpose. VVAH is a specialized pipeline with a fixed stage graph, checkpointing, and structured output contracts. It uses the same underlying models but wraps them in a disciplined workflow.
vs. other agentic SAST tools: Most are single-pass (scan → report). VVAH adds remediation (S10) and validation (S11) as integrated stages, targeting the full lifecycle from discovery to validated fix.
vs. bug bounty platforms: These rely on human researchers. VVAH automates the discovery-and-verify loop but explicitly states that findings require human review — it positions itself as a triage accelerator, not a replacement for human judgment.