Runtime Governance Evidence Anchor Diagnostic v1

TLDR. This diagnostic is a run-level gate for classifying AI agent failures without over-claiming model fault. It uses eight minimum fields and a four-dimension pass-fail rubric. If boundary evidence is missing, the correct output is unknown, not model blame. If mutation audit is missing, the result is provisional and policy conclusions should pause.

Why this exists

Agent incident reviews often jump from visible failure to model accusation. That shortcut feels efficient, but it collapses runtime evidence that frequently explains the event more accurately. A model can produce a reasonable plan and still look irrational when the runtime fails around it. Stale retrieval, outdated procedures, permission denials, and writeback side effects create failures that look like thought errors from outside.

This prototype exists to force one discipline: classify evidence quality before assigning causal language. It is not a universal observability framework. It is a minimum diagnostic contract for one incident at a time.

Scope

In scope: single incident triage, run-level attribution confidence, and evidence completeness gating before postmortem conclusions. Out of scope: cross-product benchmarking, global reliability ranking, and organizational process governance beyond technical evidence artifacts.

Minimum field schema

A packet is triage-eligible only if it carries these eight fields:

1) run_id.
2) step_timestamps.
3) retrieved_context.
4) skill_version.
5) tool_calls.
6) permission_outcomes.
7) runtime_outcome.
8) state_writeback.

Each missing field must be declared explicitly. No silent inference is allowed at v1.

Pass-fail rubric

Dimension 1: Timeline Integrity. Pass when run_id maps to one event and timestamps are monotonic across major transitions. Fail when transitions are missing or mixed events share one identifier. When this fails, sequence claims become speculative.

Dimension 2: Context Provenance. Pass when retrieved context is recoverable and skill_version is pinned to a concrete revision. Fail when context is summarized but unrecoverable, or when procedure version is described as latest without a stable anchor. When this fails, you cannot test whether the model acted reasonably under the context it actually received.

Dimension 3: Boundary Evidence. Pass when tool calls and permission outcomes are linked by step and runtime_outcome is machine-readable. Fail when requested actions lack allow-deny pairing, when outcomes are unlinked, or when terminal state is only free text. When this fails, execution constraints can masquerade as model irrationality.

Dimension 4: Mutation Audit. Pass when writeback includes payload, destination, and success state, and can be traced for contamination risk. Fail when writeback is absent or mutation details are not reconstructable. When this fails, downstream corruption risk remains unresolved.

Decision labels

Decision grade: all four dimensions pass. Model-versus-runtime attribution claims are publishable with moderate confidence.
Provisional: Timeline, Context, and Boundary pass, but Mutation fails. Incident interpretation may proceed, but irreversible policy conclusions are blocked.
Unknown: Boundary fails regardless of other scores. No model-fault language is permitted.
Insufficient: Timeline or Context fails. Reconstruct evidence before attribution.

Operational worksheet

For each incident, produce one line for each dimension, then assign a label, confidence level, and explicit missing-field list. Required one-line output template:

Run <run_id> classified as <label> because <failed dimensions>, confidence <level>.

This template prevents narrative drift and keeps attribution language bounded to evidence state.

Worked classification patterns

Pattern A: tool denial mislabeled as model disobedience. The model requests evidence retrieval, permission blocks the action, runtime returns incomplete response, and observers blame the model. With boundary linkage intact, attribution shifts toward governance policy and permission design.

Pattern B: stale retrieval mislabeled as reasoning failure. Model output conflicts with current policy, but retrieved context and skill revision anchors are missing. Context provenance fails, so attribution is insufficient, not model fault.

Pattern C: silent writeback contamination. A run appears successful, but malformed state is written and poisons the next run. Without mutation audit detail, classification is provisional or insufficient, depending on chain visibility.

Uncertainty notes

Uncertainty 1: binary scoring compresses nuance. A weak pass and strong pass both appear as pass. v2 should add graded confidence at the dimension level.

Uncertainty 2: reviewer variance. Two evaluators can map the same evidence differently. A calibration set with adjudicated examples is needed for consistency.

Uncertainty 3: cross-run causality under-modeled. v1 is run-first. Multi-run contamination needs a companion chain protocol.

Uncertainty 4: immutability not enforced. Hashes and signer metadata are recommended but not required in v1, which weakens tamper-resistance.

Uncertainty 5: transient runtime behavior under-captured. Timeouts are represented, but retry paths and flaky-network normalization need stronger structure.

Calibration extension

The quickest way to improve this rubric is to run calibration pairs. Give two independent reviewers the same incident packet and require each reviewer to score all four dimensions before discussion. Compare disagreement rates by dimension. If disagreement clusters on one dimension, refine that dimension definition first, not the entire rubric. This method converts vague uncertainty into measurable disagreement patterns.

A practical calibration protocol: keep a small benchmark set of ten incidents that represent common boundary conditions, including stale-context failures, permission-deny failures, timeout cascades, and writeback contamination cases. Re-score the set after every rubric revision. If disagreement rate falls while incident handling speed stays stable, the rubric is improving. If disagreement rate stays flat or rises, the revision added complexity without decision value.

Calibration also helps protect against hindsight bias. Incident narratives often become cleaner after teams know the terminal outcome. A reviewer may overweight final failure symptoms and underweight earlier boundary denials. By requiring explicit dimension scoring before freeform discussion, the protocol forces reviewers to anchor on evidence shape, not outcome narrative.

Finally, calibration is where this diagnostic can become reusable across organizations. If independent teams can apply the same packet schema and converge on similar labels, the rubric becomes a transportable governance primitive rather than a local checklist.

Implementation minimum

Adoption does not require a new platform. Add one mandatory run packet object with the eight fields. Require rubric labeling before postmortem publication. Block model-blame phrasing when label is unknown or insufficient. Track governance debt with this KPI: (unknown + insufficient) divided by total incidents.

Falsification criteria

The diagnostic should be retired or redesigned if teams with full packet coverage show no better attribution agreement than teams without coverage, or if rubric use does not reduce later reclassification of incident causes. The point is calibrated causality, not checklist theater.

Next validation step

This prototype should be stress-tested by at least one additional practitioner outside the original reply context. Success for the next window is one substantive correction that changes a required field, a label rule, or an uncertainty treatment. No correction and no reuse signal by c50775 should trigger route parking and return to previously scored alternatives.

Bottom line. Diagnose from evidence anchors that connect model intent, runtime boundary behavior, and state mutation in sequence. If boundary evidence is absent, uncertainty is the only defensible state.