The request timed out. Did the worker get created?

Imagine asking a provider to create a replacement worker. It creates one, but the reply never reaches your automation. The screen still says the attempt failed. Someone presses retry, and now there are two workers. Waiting longer might have spared this hypothetical team the silence. Once the silence has happened, though, a longer timeout cannot tell the next person whether the first worker exists.

The useful answer starts with an uncomfortable distinction: the caller failed to receive an answer; the provider did not necessarily fail to do the work. Reading a value again is often harmless. Asking for another resource can change the world again. Until the destination or its documented replay behavior resolves the question, the write has an unknown outcome.

Now picture the process crashing, its deadline expiring or another responder taking over. They need a durable record of the worker that was intended, not a fresh creation request. That record is the thread through the local rehearsal below; the execution contract covers the wider authority to act.

Keep the worker's identity when the request travels twice

The team wanted one approved replacement. Sending the request twice does not turn that into a wish for two. Give that intended operation an ID and save it before sending. Each transmission can have its own attempt ID for diagnosis, while retries keep the operation's parameters. A different target or effect calls for a new decision, not a quiet edit to the old request.

When repeating the operation adds no further effect, it is idempotent. The destination has to make that promise real. A UUID, a generated identifier, can name the work without preventing a single duplicate on its own. Amazon's API explanation describes how a caller's identifier lets a service distinguish a repeated transmission from an intentional request for a second resource.

Before trusting a key with the replacement worker, find out where it applies, how long it lasts, what concurrent requests do and what happens when parameters change. Those are separate questions. The word idempotent on a page, or a reminder to an AI model to be careful, does not settle them.

The details differ even within one provider. Stripe API v1 saves the first result after execution begins, including errors, compares repeated parameters and permits key removal after at least 24 hours. That is a particular interface's contract, not a property that every operational tool inherits.

Suppose its request resumes after 26 hours. A pruned key can make the replay a new request. The 24-hour minimum does not say that pruning happens exactly at that moment, and keeping a copy of the key locally does not extend the provider's protection. Check that protection still applies, or establish what happened before proceeding.

Stripe API v2 handles the same problem differently. Successful work is not performed again; failed or partially failed work can be retried, yielding an updated response or an error. Its documented replay window is 30 days for the same key and API in the same account or sandbox. The endpoint you actually call determines which promise you can use.

Leave a record the next process can pick up

Before the request leaves, save its target, normalized parameters or their digest, approval, required starting state, key and creation time in durable storage. A replacement executor can then recover the same intention. Giving the request a new identity because the old process died would make it look like new work to the destination.

There is still a gap between the provider creating a worker and the local process recording success. Durable storage does not turn those two events into one indivisible transaction. It gives the successor enough context to use a destination receipt or documented safe replay. If replay itself supplies the answer safely, a separate lookup is not obligatory.

Think of the first record as the request the team committed to, and the later success record as the result it confirmed. A crash between them leaves the account unfinished. It does not erase the worker that may already be running.

That distinction also matters when two executors wake up to the same unfinished account. Use an appropriate durable claim or concurrency control alongside the destination's protection, rather than letting both resend it as new work. A lease gives ownership for a limited time, but an old process can keep running after its lease expires. Where stale or duplicate effects must be rejected, the destination still needs to enforce that boundary.

The recovery record needs protection, too. Restrict access to identifiers and recovery details without turning the ledger into a general store of credentials. A digest can help spot changed parameters. It neither grants permission nor reliably conceals a secret with few possible values.

Find the receipt before reaching for Create again

A receipt or status endpoint tied to this exact operation is a useful first stop, where the provider offers one. If it identifies a completed resource, inspect the result the team actually needed: is that worker healthy and serving the right workload? Creation answers one question. Restoring service answers another.

An empty search page is less decisive. The new worker may not be visible yet, the responder may lack permission to see it, or the query may be looking in the wrong region. Even a fully consistent answer that it is absent now cannot prevent an earlier request, still in flight, from finishing later. Ordinary absence is not permission to create a fresh replacement.

A confirmed terminal rejection with no effect can close that uncertainty. A still-valid safe replay contract can also offer a route forward, within its documented window and the original authority. Without either, keep the unresolved operation assigned to someone. Creating a second worker is an expensive way to ask whether the first one exists.

What you can establish	What to do with the original operation
A matching operation completed	Record the result, then check the required service outcome.
The documented safe replay protection still applies	Retry the same operation within the agreed bounds.
The provider confirms a terminal rejection without an effect	Reassess the cause and current authorization before trying again.
Lookup is unavailable, merely shows absence or conflicts with the intended identity	Hold the write and leave reconciliation with a named owner.

Lose a reply where it cannot create a real worker

The Python rehearsal gives this uncertainty a small, inspectable shape. It uses Python's standard library, needs Python 3.10 or newer and makes no network calls:

python3 unknown-write-rehearsal.py

A successful run prints PASS: replay, changed intent, reconciliation holds, and naive duplicate demonstrated. That message describes assertions in the simulation, not a test of your provider.

Inside the fixture, the simulated destination saves the key and effect together, then deliberately loses the first response. Send the same key again and it returns the receipt. Change the parameters and it rejects the request. Give the work a new key and a separate case produces the duplicate that the original identity was meant to prevent.

def reconcile(status, matches_intent):
    if status == "applied" and matches_intent:
        return "verify_postcondition"
    return "hold_for_owner"

This little function does not create anything. A matching completed operation lets it move on to checking the intended outcome; otherwise it leaves the question with an owner. The other fixture cases exercise unavailable lookup and mismatched identity. None turns this single-process demonstration into a distributed exactly-once guarantee for a future provider.

The next test belongs in the provider's sandbox, with the adapter that will actually send requests. Lose a response after acceptance, restart, recover the original key and check real mismatch, concurrency and expiry behavior. The local rehearsal makes the decisions visible. It cannot establish those remote guarantees.

Give the retries one budget and the uncertainty an owner

Start the adapter with a reversible action on a disposable resource, narrowly scoped credentials and durable storage. Set both an attempt limit and an elapsed-time limit. Choose a reachable person who can inspect unresolved work. When the timer expires, the worker's whereabouts are still a question someone must answer.

Look for every place that might resend: the SDK, transport, queue and agent loop. Several sensible local retry policies can multiply into far too many transmissions. Assign one overall budget and make the other layers' behavior explicit. AWS Durable Execution guidance separates retry from replay; an at-most-once-per-retry description does not promise one effect across an entire workflow.

Keep track of how many operations remain unresolved, their age, attempts and reconciliation time, along with duplicates and failed checks of the intended outcome. Individual operation keys belong in protected logs or traces, not metric labels that create a new series for every key. You need both a manageable aggregate view and a way back to this particular worker.

Looking for the answer also uses provider quota and storage. Respect the provider's guidance and the task's deadline. If replay protection expires while an operation waits, bring it back for review rather than letting the queue automatically send it beyond the guarantee.

Stop new writes without losing the unfinished ones

A rollout stop should prevent new writes while leaving room to inspect requests already sent. If the pilot fails, disable dispatch, enumerate unresolved operations and check their destinations. Keep the ledger: deleting the record would remove the recovery identity, not undo the work.

Some effects are not reliably reversible. A recipient may already have read a message; deleted data may have no surviving copy. Where the destination cannot provide the needed guarantees, automatic retry may be unsuitable. A compensating action has consequences of its own and is not proof that the original state has returned.

The next responder should be able to find three things without reconstructing the whole shift: which worker was intended, where its outcome can be established and who owns the unanswered question. Then the timeout is unfinished work with a clear next step, rather than a second creation request waiting for a click.