☰ Contents
AISA v2.0 / Technical documentation / Failure and Recovery

Failure and Recovery

F verified factP decided planC open challenge

Purpose

Long autonomous stretches fail in the middle. A worker is evicted, a provider rate-limits, a connector drops, a session hits its limit — and the case is left in a state nobody planned. v1's answer was disk archaeology and the operator's memory F: [fact]. This document defines what the platform does instead, and names the one place where the current write protocol is unsafe by construction.

1. The invariant

P No agent turn spans an uncommitted write. A case must never depend on a database transaction staying open across a model call, a sub-agent hop, or a human gate.

Why this is a rule and not advice: the 5-step write protocol — show → apply uncommitted → verify in the DB → replay the consuming query → COMMIT last F: [protocol] — is correct discipline for a human at a terminal, and it is exactly the shape that breaks in an agent platform. Between "apply uncommitted" and "COMMIT last" there is at least one model round trip. If the worker dies there, the change is neither applied nor recorded as applied; the row locks are held until the session is reaped; the resumed case cannot tell whether its statement ran. v1 already ran into the adjacent version of this: the SQLcl MCP path is autocommit per call, so the protocol could not be executed there at all F: [rule].

P Two admissible implementations; the module chooses per operation, and the choice is recorded on the operation's capability description (§ 4):

Shape Use when
A — collapsed apply, verify and replay run inside one connector call that commits or rolls back before returning. The agent sees one result: applied-and-verified, or rolled-back-with-evidence. the verification is expressible as SQL — the common case
B — leased the connector owns the transaction under a lease with a deadline; only deterministic executor code may inspect and branch within the same session during the lease; no model call, human gate or agent hop is admitted; expiry rolls back automatically. one reviewed operation needs several deterministic apply/check steps

In both, the human's approval happens before the write opens, against the shown statement and its expected counts — not in the middle of an open transaction.

F Pre-commit verification is same-session only, and this bounds what either shape can prove (Oracle read consistency; the v1 protocol relies on it — 5-step write). An uncommitted Oracle change is invisible to every other session, so a check that runs anywhere else — the real UI, a backend service on its own pool, a second connector — cannot see it. What v1's protocol calls "replay the UI query" is the UI's SQL re-run inside the same session, not the UI itself.

P Therefore verification splits in two, and the split is part of every write plan:
Pre-commit, same session the row is as intended; counts match; the consuming query returns what the screen will render. Failure rolls back these transactional business writes; audit, trigger and external effects must still be recorded and classified
Post-commit, external surface the operator's actual screen, a backend call, a document render. This can only run after the commit — so its failure is not a rollback but a compensation, planned in advance with its own teardown (§ 2)

A plan that promises external-surface verification before commit is not implementable and does not pass H2.

2. Effects that no transaction covers

F (the PC write rules) Rollback restores rows. It does not restore: workflow instances, queue messages, generated documents, consumed numbering and BSO blanks, external registrations, cache state, or calls already made to another system.

P Every plan step declares its effect class — transactional, compensable, irreversible, or DDL:

P The DDL protocol (decision Vladimir, 02.09.2026 — D60), the full loop for CREATE OR REPLACE and friends:

  1. Capture before — the object's snapshot (content signature per the block-hash recipe, never LAST_DDL_TIME) exists in SRD_SUPPORT and the git mirror before the write opens; the signature is a precondition of the write grant.
  2. Preflight at apply — the signature is re-compared against the live object; a mismatch (someone saved in between) refuses DDL after an authorised read preflight and the case proposes a three-way merge as a new gate packet.
  3. Apply — the write executor applies the DDL under HW-ddl with the identity of the current D65 stage (D104), in one connector call; the snapshot id is recorded in the write log; STATUS per object is recorded after, so a post-apply INVALID is attributable to the apply or to the baseline.
  4. Verify — recompile check plus the dependent sweep: local dependents revalidate on first use (Oracle's work); remote dependents across a DB link do not — a spec change with @insis/@abacus dependents carries the far-side recompile as a named plan step with an assertion (the 15.05.2026 ABC_PREM incident).
  5. Compensate (H7) — re-apply the pre-apply snapshot, then re-run step 4. A failed compensation escalates to H6 with the snapshot id and the live signature — never a blind second CREATE OR REPLACE.

The standing prohibitions are the six prohibitions of Stage Planning § 3b: stale extract, spec without body, schema-wide recompile, spec with remote dependents, ungrepped callers, and snapshot overwrite.

P Teardown derives from the write log: the set removed is exactly the set written [F: artifact]. Proof that cleanup finished is evidence, not assumption — the restored state is re-read.

3. The failure cases

Failure What the platform does
Worker dies mid-apply The case's build state holds completed step ids and platform-returned ids. On resume the connector is asked what actually landed for the step in flight — the write log names the statement and its target — and the step is either recorded done or retried. Nothing is replayed blind; children re-parent onto returned ids, never predicted ones [F: the PC resumable apply].
Connector lost mid-apply The operation's effect-of-timeout contract decides (§ 4): idempotent operations are retried; non-idempotent ones become a reconciliation task — did it land? — before anything else runs. An uncertain external result is never assumed either way. WhatLandedAsync distinguishes applied, not_applied, partly_applied and unknown; a boolean found flag is insufficient.
Provider rate-limit or outage The task parks; the case does not fail. Model routes are per profile, so a fallback route is a profile decision, not a code change — but a route change is recorded, because it changes what produced the work.
Session or context exhaustion Papers and build state live in the case, not the transcript; a fresh worker resumes from them. This is the direct fix for v1's session death.
Poisoned resume Resume re-reads the papers and re-runs the assertions of completed steps it depends on; it does not trust the build state alone when the target may have changed under it.
Two cases writing the same target Impact keys — environment, system, product/version, account, route — collide and raise a coordination decision; reads never block (Agent Runtime § 11.5).
Estate caps hit Per-case budgets enforce them before the system does: INSIS allows 6 sessions/user → ORA-02391 F: [rule].

P The symptom is re-read before the fix and before the send (Support Case Studies G-6). A preflight immediately before S3 (solution take) and before S5 (application) re-reads the customer's symptom on its surface; if the customer, Bulstrad IT or another case changed it in between, the ledger records a changed outside the case event with the actor where known, and the packet is re-opened — the SD-1636 shape, where the object had been revived by Bulstrad IT while the analysis stood.

4. Connector capability descriptions

P Every connector operation declares, as data: accepted input · target identity · required context · expected effects · authorization needed · results returned · timeout behaviour · recovery properties (idempotent? compensable? verifiable after the fact?).

P The capability description is a versioned row, and its absence fails closed (D66): operations are CONNECTOR_SCOPES rows of kind operation — {operation_id, version, input_contract, target_identity, effect_class (transactional · compensable · irreversible · DDL), authorization (read grant · write grant · gate), idempotency_key shape, timeout + effect-of-timeout, recovery (retryable · reconcile-first · never-retry), evidence returned}. A connector refuses an operation that has no row — an unlisted operation is a capability gap to escalate, not a call to improvise. Registering a connector does not establish live access — availability, ownership, semantics and allowed environments are tested per adapter.

This is what makes § 3 mechanical rather than case-by-case judgement. An agent discovers which operations exist and what evidence they return; an unknown or unsupported operation is a capability gap to escalate, never a reason to improvise access.

5. Simulation

P Simulation is the default, not the exception (decision 01.09.2026). Everything is simulated first except changes that are both very small and carry no blast radius. The simulation runs on dev, QA, an isolated compose environment, or on a purpose-made new entity — venues are ranked by their co-tenants (Development module (id source) § 5 G30): an isolated compose rig first, then QA, then STAGING only with the customer's knowledge, because STAGING is customer-facing and a rehearsal there is visible; the point is that the effect is observed before it is promoted.

The cycle: capture the baseline, declare planned effects and cleanup actions, execute, observe, tear down, verify restoration. A completed simulation is not a deployment — retaining a change is a separately authorized application of it.

P Promotion is bound to confirmations and roles. A change reaches a higher environment only when its simulation evidence exists, the required gate confirmations are recorded, and the promoting actor holds the role for that target (Trust and Data § 3). Simulation evidence is part of the promotion packet, not a note in a transcript.

P The exemption test (decision 01.09.2026) — a change may skip simulation only when all five hold:

  1. a single logical object, every statement of it with an asserted expected count (a five-statement atomic orphan delete on one policy is one object; two policies are two changes);
  2. a single environment;
  3. reversible by an already-written teardown — not one that would have to be authored;
  4. no downstream consumer — nothing reads it that is not part of the same change;
  5. no customer-visible effect.

Failing any one of the five means the change is simulated. The test is applied and recorded in the plan, so the exemption is an auditable claim rather than a judgement someone made silently.

P Production simulation (decision Vladimir, 02.09.2026 — D42; a clause of the signed data-administration agreement between operator and client, D127 — customer-owned data stays the customer's and is administered per local legislation) is allowed only when all five hold: (1) the evidence cannot be obtained on any non-production target, shown by a recorded attempt or a data-dependency argument in the plan; (2) every step is transactional, or compensable with a compensation already written; (3) the blast radius is one policy, one product version or one account; (4) the pre-state is captured as evidence before the first write; (5) reconciliation is verified the same business day with a query named in the plan. It runs under a distinct gate kind H5-SIM whose role Approver(PROD, simulation) is held by two roles — the Ablera support lead and the Bulstrad system owner per system — and by no operator: four eyes by role configuration, not by platform rule (Trust and Data § 3, D107). Non-Goals N-12 stands: the exception is authorized per case, and authorizing it never moves the class.

F Measured 02.09.2026 (Measurements § DB § 6): QA cannot be restored at all today — no valid backup, NOARCHIVELOG, flashback off, no restore points, undo 900 s; the only reversal primitives are transaction rollback, the recycle bin and ≤ 15-minute flashback query. STAGING restores to last night's whole-database backup; PROD is ARCHIVELOG. P Simulation on QA therefore relies on effect inventories and teardown scripts, not on restore, until the DBA enables ARCHIVELOG and flashback and a guaranteed restore point is created before each simulation.

P Implementation contract. The deterministic A/B transaction drivers, external effect form E, durable call intents, impact claims and recovery truth table are Agent Runtime §§ 2,5,7. A lost checkpoint cannot justify repeating an effect. A dry-run performs no DML; a write-and-rollback rehearsal is a gated simulation, including a zero-row probe.

PreviousTrust and DataNextUI