Raw capture — raw/state-system.md

url: https://github.com/dbmcco/state-system date_fetched: 2026-07-05 backfilled: true

State System is a generic state substrate for agents and humans operating across many source systems.

It solves a practical agent reliability problem: an agent is often asked, "what is the current state?" while the relevant evidence is scattered across docs, messages, tasks, CRM records, calendars, repositories, memory systems, and local runtime artifacts. Some sources are fresh, some are stale, some are unreachable, some are only declared, and some are governed by another instance. Without an explicit state layer, agents either over-trust whatever context they saw last or hide uncertainty in prose.

State System makes that boundary explicit. It records what appears to be true, why that view changed, which evidence supports it, which sources are fresh or stale, what remains uncertain, which routes/tools are allowed, and which actions require governance before execution.

State is not a note, a prompt, or a transient model context dump.

State is a durable, scoped record of:

• what appears to be true now;
• why that view changed;
• which evidence supports it;
• which source systems are reachable and fresh;
• what is stale, failed, unknown, planned, or federated;
• what needs attention;
• which actions have been proposed, approved, blocked, or taken.

The model interprets meaning and proposes state transitions. Code owns schemas, evidence references, source access status, freshness checks, persistence, audit, governance boundaries, and execution safety.

This repository is the product surface. It includes:

• JSON schemas for state records, source modules, freshness, packages, routes, tool actions, federation packs, and agent-facing artifacts;
• a file-backed Python CLI runtime;
• source integration contracts for declaring connector behavior;
• preflight and freshness recorders;
• instance understanding surfaces and agent package rendering;
• package pressure tests that check whether a package exposes enough structure for an agent to answer responsibly;
• public fixtures using neutral example instances.

A deployed state root owns private runtime material: live state records, credentials, local paths, source-owned indexes, generated package exports, adapter evidence, and operational artifacts. Those do not belong in this repo.

• Declare sources.A source module says what a connector type means: source refs, access mode, preflight contract, freshness contract, index ownership, tools, read/write surfaces, gap behavior, and governance defaults.
• Declare instance capability.A deployed instance names the connectors, indexes, tools, and governance surfaces it intends to use.
• Prove access with preflight.Preflight records whether a connector is reachable.- passedproves live access;- failedand- plannedremain visible gaps.
• Prove recency with freshness.Freshness records checked time, source watermark, stale-after, lag where available, and status:- fresh,- stale,- failed,- unknown, or- planned.
• Render understanding.The runtime joins capability, preflight, freshness, indexes, and gaps into an inspectable read model.
• Render agent packages.Packages expose routes, source readiness, tool action refs, freshness gaps, evidence refs, federation packs, and governance boundaries.
• Pressure test packages.Operational questions assert that packages expose the routes, sources, gaps, freshness, tools, and federation boundaries an agent needs before answering.

Source integrations are first-class. To add one:

• Add a module to examples/source-modules/source-module-core-connectors.json.
• Give it a stable connector_type, safesource_refexamples, supported instance kinds, module modes, preflight contract, freshness contract, index contract, tool contract, gap behavior, and governance defaults.
• Add matching connector declarations to an instance capability pack or a deployed runtime capability record.
• Record preflight evidence: python3 -m state_system.cli --project-root . \ --state-root /path/to/state-root \ instance-preflight-record \ --preflight-ref preflight.state_instance.sampleco.connector.sampleco.kb \ --instance-ref state_instance.sampleco \ --connector-ref connector.sampleco.kb \ --source-ref kb:tenant:sampleco \ --connector-type kb \ --status passed \ --checked-at 2026-05-18T12:05:00Z \ --stale-after 2026-05-18T13:05:00Z \ --evidence-ref local-path:/srv/kb/sampleco
• Record freshness evidence: python3 -m state_system.cli --project-root . \ --state-root /path/to/state-root \ instance-source-freshness-record \ --instance-ref state_instance.sampleco \ --connector-ref connector.sampleco.kb \ --source-ref kb:tenant:sampleco \ --connector-type kb \ --status fresh \ --checked-at 2026-05-18T12:05:00Z \ --source-watermark kb.indexed_at:2026-05-18T12:04:00Z \ --watermark-basis source_index \ --latest-indexed-at 2026-05-18T12:04:00Z \ --freshness-policy-ref source_module.folio.freshness \ --stale-after 2026-05-18T13:05:00Z \ --evidence-ref freshness:kb:fresh:20260518T120500Z``checked_atproves the adapter ran.watermark_basisand the typed latest timestamps say whether the corpus or index is current. Useprobe_onlywhen the adapter checked availability but cannot prove corpus/index recency.
• Declare source-owned index refs when retrieval exists. State System may cite source indexes without owning raw corpora.
• Add adapter commands to a fleet freshness manifest when the source can be refreshed mechanically.
• Rebuild/export the instance agent package and add package pressure questions for answer paths affected by the source.

Detailed docs:

• docs/source-modules.md
• docs/runbooks/open-source-onboarding.md
• docs/runbooks/fleet-freshness-runner.md

Agents should consume rendered artifacts, not scrape private runtime roots. Start with AGENTS.md and docs/agent-integration.md.

The short contract:

• read rendered InstanceAgentPackageartifacts orinstance-agent-packages-read-model.json;
• inspect source readiness, preflight, freshness, stale-after expiry, source gaps, route gaps, and federation gaps before answering;
• use explicit question routes, source module refs, tool action refs, and federation pack refs;
• do not infer source behavior from connector names;
• do not materialize raw federated data unless a pack explicitly permits it;
• treat captured agent output as evidence for review, not accepted truth;
• keep governance separate from freshness and preflight.

From a clean checkout:

python3 -m unittest discover -s tests
python3 -m state_system.cli --project-root . validate
python3 -m state_system.cli --project-root . report-suite-run --output-dir /tmp/state-system-report-suite

Open /tmp/state-system-report-suite/index.html to inspect the generated reports. The validation command checks shipped schemas and JSON examples; the test suite checks runtime, package, freshness, federation, source-module, and pressure-test contracts.

Run a trace:

python3 -m state_system.cli --project-root . trace-run examples/traces/linear-deal-won.trace.json --output-dir /tmp/state-system-traceBuild a North Star answer substrate from packages:

python3 -m state_system.cli --project-root . north-star-answer \
  --query "What is the current state?" \
  --package sample=examples/instance-agent-package/instance-agent-package-sample-personal-nova.json \
  --output-dir /tmp/state-system-north-star

Render the deterministic text view:

python3 -m state_system.cli --project-root . north-star-answer-render \
  /tmp/state-system-north-star/north-star-answer.json \
  --check \
  --output-path /tmp/state-system-north-star/north-star-answer.txt

The shipped fixtures use neutral example organizations and instances such as SampleCo, ResearchCo, PortfolioCo, and sample_personal. They are contract fixtures, not required deployment names. Real deployments should use their own instance refs, connector refs, source refs, package IDs, and freshness manifests.

• state_system/- Python runtime and CLI implementation.
• schemas/- JSON schemas for state, packages, freshness, routes, and tools.
• examples/- public schema-valid fixtures and trace manifests.
• tests/- unit, conformance, runtime, and pressure tests.
• docs/NORTH_STAR.md- intended direction and product boundary.
• docs/system-diagram.html- local architecture diagram.
• docs/source-modules.md- source connector extension contract.
• docs/agent-integration.md- runtime agent consumption contract.
• docs/runbooks/- public operator runbooks.

Run these before publishing or merging:

python3 -m unittest discover -s tests
python3 -m state_system.cli --project-root . validate
python3 -m unittest tests.test_open_source_ecosystem_conformance -v
git diff --check