Timed-out automation writes: reconcile before retrying

A timeout leaves the caller with a missing answer. It does not tell the caller whether the destination changed. For an operational agent, that difference determines whether the next attempt recovers a task or repeats a consequential action.

Consider a hypothetical automation that creates a replacement worker. The provider creates it, but the response is lost. The agent interprets the timeout as failure and submits a new creation request. Now there are two workers, while the task history still describes one unsuccessful attempt. A longer timeout might make the incident less likely; it would not resolve the ambiguity when it happens.

Keep a timed-out write in an explicit unknown state until destination evidence or a documented idempotency contract resolves it. This guide develops that rule from our production-agent execution contract into a small implementation and rehearsal plan. The aim is to make the retry decision inspectable even when the original agent process is gone.

An operation can have several attempts

An operation represents the intended effect: create this replacement worker for this approved recovery. An attempt is one transmission of that request. Give the operation a stable identifier before sending it; give each attempt its own identifier for diagnosis. A retry must retain the operation identity and intended parameters. A changed target or changed effect needs a fresh decision.

An idempotent operation can be repeated without adding to its intended effect. An idempotency key helps only when the destination implements a contract for that key. Generating a UUID locally and putting it in a log does not prevent duplicate resources. Amazon’s explanation of idempotent APIs discusses caller-provided request identifiers, including the distinction between a repeated request and a caller deliberately requesting two similar resources.

Read the destination’s precise rules. What scopes the key: account, endpoint, region or something else? How long does the service retain it? What happens if parameters change, two attempts overlap or the first attempt returns an error? Those answers belong in the tool adapter rather than in an instruction asking the language model to retry carefully.

For a concrete example of differing semantics, Stripe documents storing the first result after execution begins, including error results, comparing repeated parameters, and allowing keys to be removed after they are at least 24 hours old. This is an illustration of why the contract matters, not a universal policy for operational APIs.

Persist intent before sending the write

A production implementation needs a durable operation record containing the target, normalized parameters or their digest, authorization reference, relevant state precondition, operation key and creation time. Save that record before dispatch. If the process crashes after sending, its replacement must recover the original identity instead of inventing another one.

The dangerous gap is between a remote side effect and the local acknowledgement. A local database transaction cannot make a remote API call atomic with the local record merely because both occur inside one function. Design the recovery path to revisit an operation whose transmission or acknowledgement is uncertain.

Prevent two executors from independently deciding to submit the same unresolved operation. Use the destination’s duplicate protection where supported and an appropriate durable claim or concurrency control in the executor. A lease alone needs careful handling when an old worker continues after expiry; the destination must still reject stale or duplicate effects where that guarantee is required.

Keep credentials and private payloads out of general audit logs. Store only the identifiers and evidence needed to reconstruct the decision, with access appropriate to the resource. A digest can help detect changed parameters, but it does not authorize the operation or conceal every low-entropy secret.

Ask what the destination can prove

Prefer a documented operation-status endpoint or a provider receipt tied to the exact operation. If the provider confirms completion, record the resource identity and inspect the intended postcondition. A worker that exists may still be unhealthy or attached to the wrong workload. Execution completion and service recovery remain separate observations.

An empty resource search is weaker evidence. The result may be delayed, filtered by permissions or scoped to a different region. Even a strongly consistent absence at one instant does not prove that an earlier in-flight request cannot complete later. Do not convert an ordinary “not found” response into permission for a fresh write.

A provider may instead document a terminal rejection that guarantees no effect, or a safe replay using the original idempotency key. Use that exact contract, within its retention window and the original authorization. If neither route exists, keep the operation unresolved and hand it to a named owner. The cost of waiting is visible work; a speculative second effect can be harder to undo.

Evidence	Decision
Matching operation completed	Record its result and check the service postcondition.
Documented safe replay contract still applies	Retry the same operation within bounded policy.
Provider proves terminal rejection without an effect	Reassess the cause and current authorization before another attempt.
Lookup unavailable, ordinary absence, or conflicting identity	Hold the write and retain an owner for reconciliation.

Rehearse the lost response locally

The downloadable Python rehearsal uses only the standard library and makes no network calls. Run it with Python 3.10 or newer:

python3 unknown-write-rehearsal.py

The fixture simulates a destination that atomically records an operation key with its effect. It deliberately loses the first response after applying the change. Replaying the same key returns the existing receipt; changing the parameters under that key fails. A second scenario shows how a naive retry with a new key creates a second effect.

def reconcile(status, matches_intent):
    if status == "applied" and matches_intent:
        return "verify_postcondition"
    return "hold_for_owner"

This small function is intentionally conservative. It does not issue writes and does not turn missing evidence into a retry. The complete fixture also exercises unavailable lookup and a mismatched operation. Its passing assertions demonstrate these local cases, not a distributed exactly-once guarantee.

Use the example to start an adapter test, then replace the simulated destination with your provider’s sandbox. Inject a lost response after the destination accepts a write, restart the executor and confirm that the original operation key survives. Test the provider’s actual parameter-mismatch, concurrent-request and expired-key behavior. Do not deliberately fault a production control plane to run this exercise.

Integrate one consequential tool

Start with one reversible action in a nonproduction environment, such as creating a disposable test resource. You need a provider sandbox, narrowly scoped credentials, a durable store and a reachable owner who can inspect destination state. Agree on an attempt limit, elapsed-time limit and the conditions requiring an owned hold before enabling retries.

Inventory retries already present in the SDK, transport, queue and agent loop. If each layer retries independently, one task can create many transmissions. Choose which layer owns the overall attempt budget and make the other layers’ behavior explicit. The AWS Durable Execution guidance distinguishes replay behavior from retries; a label such as at-most-once per retry does not mean an entire workflow can never repeat a side effect.

Measure unresolved operation count and age, attempts per operation, reconciliation latency, duplicate effects and failed postconditions. Keep the operation key out of high-cardinality metric labels; retain it in controlled logs or traces for investigation. The useful improvement is fewer unexplained or duplicate effects at an acceptable completion delay, not simply a lower timeout count.

Budget the reads and storage as well as the writes. Reconciliation can consume API quota, and long retention increases durable-state cost. A retry delay should respect provider guidance and the task deadline. If the provider’s duplicate-protection window ends before your workflow can safely resume, an old queued task needs review rather than automatic replay.

Stop with the unfinished work still owned

Roll out the adapter behind a control that stops new dispatches while preserving reconciliation of in-flight operations. On a failed pilot, disable additional writes, enumerate unresolved operations and verify their destination state. Deleting the operation ledger would remove the evidence needed to recover.

Some effects have no reliable undo: an external notification has already been read, or deleted data has no recoverable copy. For those actions, missing destination guarantees may rule out automatic retries entirely. A compensating action can have its own consequences and must not be presented as restoring the original state.

Take one existing tool timeout and answer three practical questions in its runbook: which durable record identifies the original operation, which destination observation can resolve it, and who owns it when that observation is unavailable. If one answer is missing, keep the write on hold until the execution path can supply it.