9 Executable verification and machine evidence

Quarto-Needs distinguishes three claims that are often collapsed into one:

a requirement states an engineering obligation;
a modeled test-case states how that obligation is verified;
machine evidence records what executable test actually ran and whether its linkage agrees with the engineering model.

This distinction prevents a green test suite from being treated automatically as proof for an unrelated requirement.

9.1 Link pytest tests to engineering objects

The Quarto-Needs pytest plugin provides two markers:

import pytest

@pytest.mark.requirement("FUN-004")
@pytest.mark.quarto_need_test_case("TC-010")
def test_graph_exploration_assets():
    ...

requirement identifies the engineering obligations claimed by the executable test. quarto_need_test_case binds the pytest node to one or more modeled test-case objects.

The model should carry the reciprocal executable binding:

::: {.need #TC-010 type="test-case" status="passed"
      pytest-nodeid="tests/test_graph.py::test_graph_exploration_assets"
      evidenced-by="EVD-010"}
## Interactive graph exploration regression
...
:::

The stable pytest node ID is a physical execution reference. It does not replace the modeled test-case, which remains the engineering verification object.

9.2 Generate deterministic provider evidence

Writing an artifact is opt-in. Ordinary pytest runs have no Quarto-Needs file side effects.

pytest \
  --quarto-needs-evidence=.quarto-needs/evidence/pytest-provider.json

The same destination can be supplied with QUARTO_NEEDS_PYTEST_EVIDENCE.

The version-1 provider artifact is validated by schemas/evidence-pytest-v1.schema.json and contains deterministic semantic execution data:

{
  "provider": "pytest",
  "schemaVersion": "1",
  "tests": [
    {
      "nodeid": "tests/test_graph.py::test_graph_exploration_assets",
      "outcome": "passed",
      "requirements": ["FUN-004"],
      "testCases": ["TC-010"]
    }
  ]
}

Evidence v1 intentionally exposes only passed, failed, and skipped. Pytest’s richer expected-failure semantics are normalized conservatively: an expected XFAIL is recorded as skipped, while any XPASS is recorded as failed, including a non-strict XPASS that pytest itself may report as a passing test. An unexpected pass therefore cannot become positive engineering evidence silently.

Timestamps and durations are intentionally excluded from the provider payload so identical engineering results remain reproducible.

9.3 Provider-neutral attestation envelope

Execution provenance belongs to a separate provider-neutral envelope defined by schemas/evidence-envelope-v1.schema.json. This preserves deterministic provider output while binding that output to a concrete engineering state.

Create an attestation from a provider payload with:

quarto-needs --root . evidence attest \
  .quarto-needs/evidence/pytest-provider.json \
  --output .quarto-needs/evidence/pytest.json \
  --expires-hours 24

--source-revision may be supplied explicitly. In GitHub Actions, GITHUB_SHA is captured automatically when no explicit revision is provided.

An envelope records:

provider name and version;
generatedAt and optional expiresAt timestamps;
optional source revision;
the current configuration, semantic-graph, and representation fingerprints;
the provider artifact schema;
a canonical SHA-256 digest of the embedded provider payload;
the provider payload itself.

Conceptually:

{
  "schemaVersion": "1",
  "kind": "quarto-needs-evidence",
  "provider": {"name": "pytest", "version": "8.0"},
  "generatedAt": "2026-08-30T12:00:00Z",
  "expiresAt": "2026-08-31T12:00:00Z",
  "subject": {
    "sourceRevision": "abc123",
    "configurationFingerprint": "...",
    "semanticGraphFingerprint": "...",
    "representationFingerprint": "..."
  },
  "artifact": {
    "schema": "evidence-pytest-v1",
    "digest": "sha256:...",
    "payload": {"provider": "pytest", "schemaVersion": "1", "tests": []}
  }
}

The envelope is intentionally provider-neutral. Pytest is the first executable provider, while evidence-checks-v1 provides a common deterministic payload for JUnit XML, coverage.py, Quarto render, JSON Schema validation, lint, and type-check adapters.

The payload digest is checked before semantic validation. A modified payload therefore cannot keep using an old attestation silently.

9.4 Validate evidence against the current model

Generation is only half of the contract. Validate either a raw provider artifact or an attested envelope against the canonical engineering graph:

quarto-needs evidence check .quarto-needs/evidence/pytest.json

For machine consumers:

quarto-needs evidence check \
  .quarto-needs/evidence/pytest.json \
  --format json

The command remains backward compatible with raw evidence-pytest-v1. It also understands raw/attested evidence-checks-v1. When an attestation is supplied, provenance is validated against the current snapshot.

The pytest semantic check verifies that:

every executable test passed;
every evidence entry is bound to a modeled test-case;
every referenced test-case exists;
its modeled pytest-nodeid equals the executable node ID;
every referenced requirement exists;
every requirement claimed by the executable test is actually connected by a verification-family relation to a bound modeled test-case;
reciprocally, every requirement that the model says is verified by a bound modeled test-case is claimed by the executable test;
every modeled test-case carrying pytest-nodeid is present in the evidence artifact.

For attested evidence it also verifies that:

the configuration fingerprint still matches;
the semantic graph fingerprint still matches;
the representation fingerprint still matches;
generatedAt is a valid, non-future timestamp;
an explicit expiresAt, when present, has not expired and does not precede generatedAt;
when GITHUB_SHA is available, an attested sourceRevision agrees with the current CI revision.

A successful artifact therefore supports the chain in both directions and ties the executable result to the graph state it was meant to prove:

requirement --verified-by--> modeled test-case
                              │
                              │ execution binding
                              ▼
                         executable provider
                              │
                              ▼
                    deterministic provider payload
                              │ SHA-256 + graph fingerprints
                              ▼
                   provider-neutral attestation

9.5 Evidence diagnostics

EVD001 precedes all of them: it reports a pytest artifact whose tests array is missing or malformed. Because no check could be interpreted, no EVD1xx diagnostic is produced alongside it.

Provider-specific pytest diagnostics use stable EVD1xx codes:

Code	Meaning
`EVD101`	executable outcome is not `passed`
`EVD102`	executable test has no modeled test-case binding
`EVD103`	evidence references an unknown test-case
`EVD104`	referenced modeled test-case has no `pytest-nodeid`
`EVD105`	modeled and executable pytest node IDs disagree
`EVD106`	evidence references an unknown requirement
`EVD107`	executable test claims a requirement not verified by a bound modeled test-case
`EVD108`	a modeled pytest-bound test-case is missing from the artifact
`EVD109`	a bound modeled test-case verifies a requirement that the executable test does not claim

Provider-neutral attestation diagnostics use EVD2xx:

Code	Meaning
`EVD201`	attestation has no valid subject
`EVD202`	configuration fingerprint is stale
`EVD203`	semantic graph fingerprint is stale
`EVD204`	representation fingerprint is stale
`EVD205`	attested source revision differs from the expected revision
`EVD206`	`generatedAt` is invalid
`EVD207`	evidence claims to have been generated in the future
`EVD208`	evidence exceeds a caller-supplied maximum age
`EVD209`	`expiresAt` is invalid
`EVD210`	evidence has expired
`EVD211`	`expiresAt` precedes `generatedAt`

Generic machine-check validation uses EVD3xx; see Machine evidence providers.

EVD107 and EVD109 form the reciprocal traceability gate: together they require agreement between executable markers and the model’s verification relations rather than accepting either source as authoritative on its own.

Malformed, unsupported, or digest-invalid evidence artifacts are usage/artifact errors and exit with code 2. I/O failures exit 3. A structurally readable artifact that disagrees with the engineering model or whose attestation is stale exits 1.

9.6 Self-hosted example

The self-hosted Quarto-Needs case study exercises the complete flow directly:

make evidence-self-example

That target:

executes representative real tests for ADR governance, public graph safety, named graph views, and impact analysis;
writes the deterministic provider artifact to examples/quarto-needs/.quarto-needs/evidence/pytest-provider.json;
creates a 24-hour attestation at examples/quarto-needs/.quarto-needs/evidence/pytest.json, including GITHUB_SHA automatically in CI;
validates the attestation against the current self-hosted engineering graph.

make render-self-example depends on this target, so the example cannot render successfully with stale fingerprints, an expired attestation, a mismatched CI revision, or inconsistent executable bindings.