Recovering an automation write after a timeout

Preserve a timed-out write as explicitly unknown until destination evidence or the destination’s documented idempotency contract resolves it. The production-agent execution contract supplies that rule; this guide follows the worker-creation example into durable operation identity, a small local rehearsal and a tool adapter. The aim is a recovery decision another process or person can inspect even after the original executor disappears.

A timeout can leave one system saying “no answer” while another already holds the new resource. For a read, repeating the request may be straightforward. For a write, the same reflex can create a second effect before anyone has established the first. The recovery problem is therefore a question about destination state, not merely how long the caller should wait.

Consider a hypothetical automation creating a replacement worker. The provider creates it, but the response disappears. The agent sends another creation request and produces a second worker, while its history still describes the original attempt as unsuccessful. Increasing the timeout might reduce the frequency of this sequence. It cannot resolve the uncertainty when a response is lost after an effect occurs.

One intended worker, several possible attempts

Separate the intended effect from the network transmissions used to achieve it. “Create this replacement worker for this approved recovery” is one operation; sending its request is an attempt. Persist an operation identifier before the first attempt and keep separate attempt IDs for delivery diagnosis. Retrying must retain the same intended parameters and operation identity. Choosing another target or effect creates a different decision rather than a retry of the first one.

Idempotence means repetition does not add to the intended effect. A key only provides that property when the destination implements it. A locally generated UUID in an audit log helps identify work but cannot stop a provider from creating duplicate resources. Amazon’s explanation of idempotent APIs explains how caller-provided identifiers distinguish repeated transmission from an intentional request for two otherwise similar resources.

The adapter therefore needs the provider’s actual contract: the account, endpoint or region in which a key applies; its retention period; and the handling of changed parameters, overlapping attempts and errors. Encoding those rules gives recovery a defined path. A prompt telling the model to be careful leaves the network and destination semantics unchanged.

For example, Stripe’s API v1 documentation describes retaining the first result after execution begins, including error results, comparing repeated parameters, and permitting key removal after at least 24 hours. Those are specific documented semantics, not a policy to assume for every operational API. Reading the contract tells the adapter what a repeated key will really buy.

Consider a hypothetical executor resuming 26 hours after that first Stripe API v1 request. If Stripe has already pruned the key, replaying it starts a new request rather than retrieving the saved result. The 24-hour minimum does not establish that pruning happens at exactly that age. Persisting the key is necessary, but the adapter must also establish that the documented protection still applies or reconcile the destination before resuming. Keeping the same identifier does not extend its guarantee.

Stripe API v2 has a different replay contract: successful work is not repeated, while failed or partially failed work is retried when possible, returning an updated response or an error if the retry cannot succeed. Its documented replay window is 30 days for the same key, API and account or sandbox. An adapter must select the contract for the endpoint’s namespace rather than carry the API v1 saved-error and retention assumptions into v2.

The process can crash after sending and before recording success

Store the operation’s target, normalized parameters or digest, approval reference, relevant state precondition, key and creation time durably before dispatch. A replacement executor must recover that record rather than manufacture a new identity. Without it, even a destination that supports safe replay may receive what appears to be a brand-new operation.

In the worker example, the remote resource can exist while the local record remains pending. Persisting identity makes that mismatch recoverable, but does not atomically join the remote call and local acknowledgement. Placing both in one function cannot close the distributed gap. Destination inspection remains necessary to learn which side completed before the crash.

Concurrency introduces another version of the same problem. Two executors must not independently decide to send an unresolved operation as new work. Combine supported destination duplicate protection with an appropriate durable claim or concurrency control. A lease requires special care when an old worker keeps running after expiry; where stale or duplicate effects must be rejected, the destination still needs to enforce that guarantee.

Keep this record useful without turning a general audit log into a credential store. Save identifiers and recovery evidence under resource-appropriate access controls, excluding secrets and private payloads. A parameter digest can expose a mismatch, but neither authorizes the write nor reliably conceals every low-entropy secret.

A missing search result is weaker than a matching receipt

Begin reconciliation with an operation-status endpoint or provider receipt tied to the exact operation, when available. A confirmed completion supplies the resource identity and allows the next check: did it achieve the intended postcondition? The created worker may be unhealthy or attached to the wrong workload. Creation is evidence of the effect, not yet of successful recovery.

An ordinary empty resource search establishes much less. Visibility can lag, permissions can filter results, and region scope can be wrong. Even strongly consistent absence at one moment does not prevent an earlier in-flight request from completing afterward. Without a stronger destination guarantee, “not found” is not permission to create a second operation.

Other documented contracts may resolve the uncertainty: a terminal rejection that guarantees no effect, or safe replay with the original idempotency key. Use the exact contract within its retention window and original authorization. When neither is available, retain the unresolved operation with a named owner. Waiting has a visible cost, but another speculative effect may create a recovery problem that is much harder to reverse.

Evidence	Decision
Matching operation completed	Record its result and check the service postcondition.
Documented safe replay contract still applies	Retry the same operation within bounded policy.
Provider proves terminal rejection without an effect	Reassess the cause and current authorization before another attempt.
Lookup unavailable, ordinary absence, or conflicting identity	Hold the write and retain an owner for reconciliation.

Watch a deliberately lost response in a local fixture

The downloadable Python rehearsal uses the Python standard library, makes no network calls and runs with Python 3.10 or newer:

python3 unknown-write-rehearsal.py

Its simulated destination records an operation key atomically with the effect, then deliberately loses the first response. Sending the same key again returns the existing receipt; changing parameters under that key fails. A separate case retries with a new key and produces another effect, exposing the precise difference between another attempt and another intended operation.

def reconcile(status, matches_intent):
    if status == "applied" and matches_intent:
        return "verify_postcondition"
    return "hold_for_owner"

The displayed reconciliation function never writes. Only a matching applied result advances to postcondition verification; other states wait for an owner. The full fixture also covers unavailable lookup and mismatched operations. Its assertions make those local decisions inspectable, without claiming a distributed exactly-once guarantee for a real provider.

Use the fixture as the beginning of an adapter test, then substitute the provider’s sandbox. Lose a response after an accepted write, restart the executor and verify recovery of the original key. Exercise actual mismatch, concurrent-request and expired-key behavior rather than inferring it from the fixture. A nonproduction sandbox supplies these failure cases without deliberately disrupting a production control plane.

The adapter coordinates retries and reconciliation

Start with a reversible action such as creating a disposable resource in a provider sandbox. Narrow credentials and a durable store make the test bounded. Agree on attempt and elapsed-time limits, plus a reachable person who can inspect destination state when the adapter cannot settle it. That owner is part of the recovery path, not a name added after the timeout.

Inspect retries already implemented by the SDK, transport, queue and agent loop. Independent retry layers can multiply transmissions from one task. Assign the overall attempt budget to one layer and make the others explicit. The AWS Durable Execution guidance separates replay from retries; an at-most-once-per-retry label does not imply that an entire workflow can never repeat an effect.

Follow unresolved operations to their eventual decisions. Count and age, attempts per operation, reconciliation latency, duplicate effects and failed postconditions show whether the adapter reduces uncertainty at an acceptable delay. Put individual operation keys in controlled logs or traces instead of high-cardinality metric labels, preserving both aggregate visibility and a path to investigate a particular worker.

Reads and durable storage have costs too. Reconciliation uses API quota, retention consumes storage, and delay needs to respect provider guidance and the task deadline. If a duplicate-protection window will expire before safe resumption, an old queued operation needs review. Replaying it automatically after the guarantee has elapsed silently changes the risk.

Stopping dispatch should leave reconciliation alive

Use a rollout control that blocks new writes while allowing inspection of in-flight operations. If the pilot fails, disable dispatch, enumerate unresolved work and check its destination state. Preserve the ledger: deleting it removes the identities needed to recover rather than undoing anything at the provider.

Some effects cannot be reliably reversed. A notification may already have been read, or deleted data may lack a recoverable copy. Without suitable destination guarantees, such actions may rule out automatic retry entirely. A compensating action introduces its own consequence; it is not proof that the original state has been restored.

For one existing tool timeout, put three concrete references in the runbook: the original durable operation record, the destination evidence that can settle its outcome, and the owner who handles unavailable evidence. A replacement process can then resume the investigation of the intended worker instead of accidentally requesting a second one. Where one of those references is missing, keep the write on hold until the recovery path is understood.