When an automated restart gets no reply

The restart command goes out. No reply comes back. In this hypothetical repair, the worker might already be restarting, or the request might never have arrived. Sending another command could interrupt recovery; doing nothing could leave the service stuck. Before trying again, someone needs to discover what the destination actually did.

That is the part of automated remediation an alert-to-command diagram can miss. Software must choose an appropriate repair, limit its effects, follow its outcome and hand unresolved work to a person. A familiar command is only one step in that job.

Automated remediation means software performs the repair, rather than only reporting a fault or recommending an action. A fixed rule and runbook can do that for well-understood failures. AI may help interpret context or propose a response; a deterministic executor checks whether the particular action is permitted against this target in its current state.

Turn the proposal into a checkable operation

Have the model identify an approved, versioned runbook and parameters the executor can validate. An unrestricted shell command gives the executor far less to check. Recheck eligibility immediately before acting, because the incident may have changed while the proposal was prepared.

For our hypothetical restart, the executor first needs to know which worker is unhealthy now. It connects that observation to the service, recent changes and dependency state. If the queue is blocked by a shared dependency, a worker restart may not fit the approved runbook at all. With no established response that fits, the supported diagnosis goes to a responder, not to a newly invented command.

Even when the runbook fits, taking this worker down needs spare capacity and no conflicting work. The executor checks the target scope and required approval against the current state. A confident diagnosis cannot supply missing permission. If another repair has already consumed the allowed capacity, this proposal must wait or go to its owner rather than quietly become a second restart.

Only a permitted operation proceeds with a scoped identity, a shared operation record, a target limit and a deadline. That record must remain useful after the command leaves: the workflow will need independent recovery evidence, and a named owner must receive failure or uncertainty. The worker being eligible for a restart is not evidence that the restart fixed it.

For example, AWS Systems Manager's approval step can pause an automation for a manual decision. The wider incident-management workflow still needs to say who receives the work when approval or execution cannot proceed.

Choose supervision for the repair you actually have

Keep a new failure mode, uncertain diagnosis or untested repair recommendation-only. Human-approved execution fits an established runbook when a person still needs to judge its target, customer effects or limits. Limited automatic execution requires machine-checkable preconditions, bounded consequences, independent recovery measurements and an exercised stop path.

That permission must be reversible. A restart tested before a dependency migration may be unsuitable afterward even if the alert keeps its old name. Return to supervision when the evidence supporting automatic operation no longer applies.

Restart one worker, with a reason to stop there

For our worker, set the maximum affected targets, the wait before another action and conditions that forbid repetition. If a shared dependency is blocking the queue, repeated worker restarts may add load while removing capacity. The first operation should teach the workflow whether more action would help.

A lost response interrupts that reasoning. First establish whether the command happened. Command acceptance, command completion and service recovery are three different observations. None can safely stand in for all three.

Keep an uncertain write open until it is reconciled

Give the operation an identifier so the executor can ask the destination about that exact restart, then inspect the worker's current state. Where the destination supports an idempotent API, repeating the same identified request can avoid multiplying the effect. Calling an ordinary restart “idempotent” in your own record does not change how the destination handles it.

Coordinate across executors too. Two executors each allowed to restart one worker can collectively remove two workers. An executor that forgets an unresolved request after its own restart can repeat the same mistake. Use a shared record or suitable concurrency mechanism to enforce the service's total limit, not merely each caller's local limit.

Some consequences cannot simply be reversed. A restart may leave a message delivered or a payment submitted. The runbook should say when to stop, which records reveal those effects and who can resolve them. An uncertain write remains unknown until evidence reconciles it.

Ask what the customers and queue can do now

Suppose the destination now confirms that the worker restarted, but queue age is still rising. We have recovered the missing command result, not established service recovery. Choose the service condition before the pilot: it might be falling queue age, resumed successful transactions or recovery of the affected request path. Then keep watching that condition rather than switching to whichever chart looks encouraging.

The next responder should be able to see why this worker was selected, which eligibility checks permitted the exact request and what the destination has since confirmed. Beside those facts, the queue observation shows what remains unresolved. Give that wait a deadline and an escalation owner; “restarted but recovery not established” calls for a different next step from “recovered” or “the command outcome is still unknown.” Keep failures distinct from both.

If this restart was unnecessary or did not help, keep it in the pilot record. Was the worker eligible for the wrong reason, did the action remove too much capacity, or did the chosen recovery signal mislead us? Those attempts can reveal what needs changing before a recommendation-only repair earns supervised or automatic execution.

Command success still needs a service check

Read diagram description

Illustrative bounded remediation: establish current eligibility, coordinate one operation, and check an independent user signal. An ambiguous write result requires reconciliation; failed recovery requires a stop or escalation, not an uncontrolled retry loop. Diagram labels: Current eligibility: Target unhealthy, spare capacity, no conflicting action; One bounded operation: Shared operation record, target limit and cooldown; Destination result: Reconcile any timeout before another write; Independent service check: Queue age, user transactions and adverse effects; Verified recovery: Record supporting evidence; Failed or inconclusive: Stop and escalate with the record.

Keep all forty incidents in the account

Suppose recovery for the worker requires queue age below an agreed threshold within a defined window, without increased failed jobs. Let the workload determine those values. Follow every eligible incident, not just the appealing repairs.

In an illustrative pilot, 40 incidents are eligible, automation attempts 30 and verifies recovery in 24. Recovery among attempts is 24 / 30 = 80%. Verified automated coverage is 24 / 40 = 60%. Keep both denominators: six attempts remain unverified and ten eligible incidents had no attempt.

Separate observed failures from unknown outcomes. Track unnecessary actions, harmful changes, recurrences, manual follow-up and time from detection to verified recovery. Compare similar incident types and keep unsuccessful attempts in duration accounting. The KPI measurement guide explains the denominator and window choices.

Carry the tested limits into the runbook

Use the incident runbook template to record connected decisions:

• Scope: failure mode, service owner, targets, exclusions and fresh eligibility evidence.
• Authority: identity, permitted operation and approval policy for this state.
• Limits: shared operation record, maximum targets, cooldown and deadline.
• Outcome: independent success condition, observation window and adverse effects.
• Stop: timeout and failed-recovery handling, unavailable rollback, audit record and the person who can disable the workflow.

For this restart, approval depends on current worker health, spare capacity and no conflicting action. It is permission for a bounded operation against a defined state, not permanent permission to use a command. A migration, concurrent change or monitoring loss can remove that basis and require supervision again.

Rehearse the lost reply before leaving the repair unattended. Another person should be able to find the exact operation, discover what the worker did and continue without restarting blindly. Then an interrupted automation leaves behind a repair someone can finish, rather than a green light nobody can explain.