Automating a worker restart from diagnosis to recovery

Consider a hypothetical automated restart that sends its request and then loses the response. The worker may already be restarting. Sending the command again could interrupt recovery or repeat side effects, while doing nothing may leave the service unhealthy. The next useful step is to find out what happened at the destination.

This awkward middle state explains much of the design work in automated remediation. The workflow must select a suitable repair, limit what it can change and establish whether the service recovered. AIOps can help connect detection and diagnosis to the repair. A deterministic executor can apply the same eligibility rules and limits regardless of how confidently the model recommends it.

A frequent, understood failure is a useful starting point because the team can define those rules and inspect repeated outcomes. Following the restart through the complete workflow shows what needs to be in place before the repair can run unattended.

From a diagnosis to a versioned runbook

The model's output should identify an approved, versioned runbook and parameters the executor can validate. That gives the execution layer something specific to permit or reject instead of an unrestricted shell command. Because the incident may change while the proposal is prepared, the executor must check current eligibility just before acting.

1. Gather the incident context. Associate the service, health signals, recent changes and dependency state with the proposed repair so its rationale can be inspected.
2. Select an established response. Match that context to an approved runbook. When none fits, preserve the diagnosis for a responder rather than invent an executable repair.
3. Check whether execution is permitted now. Evaluate target scope, spare capacity, conflicting work and any required approval. Confidence in the diagnosis does not supply permission.
4. Record and limit the operation. Use a scoped identity, one operation record, a target limit and a deadline so the action can be followed and contained.
5. Observe the service afterward. Compare independent signals with the recovery condition and give failure or uncertainty an explicit stop and escalation route.

AWS Systems Manager offers an approval step that pauses an automation for a manual decision as one concrete approval mechanism. The surrounding AI incident-management workflow still needs to identify who receives the work when that mechanism or the repair cannot proceed.

How much supervision does this action need?

Recommendation-only operation fits a new failure mode, uncertain diagnosis or untested repair. Human-approved execution becomes useful when the runbook is established but its target, customer effect or recovery limits require a person to choose. Limited automatic execution calls for machine-checkable preconditions, bounded consequences, independent recovery measurements and an exercised stop path.

These are decisions about the action in its present context. Unexpected effects or a material dependency change can invalidate the evidence for automatic execution and justify returning to supervision. The workflow should be able to become more cautious as well as more autonomous.

What one worker restart can tell you

For the worker restart, specify the maximum affected targets, the interval before another action and the conditions that forbid another attempt. A queue blocked by a shared dependency may gain nothing from more restarts; they can add load while consuming capacity. The first bounded operation should therefore produce evidence about whether continuing is useful.

The lost response introduces a separate question. Before judging whether another restart is appropriate, establish whether the first one happened. Command acceptance, command completion and service recovery are different observations, and each answers a different part of the decision.

A lost response leaves the operation unresolved

The operation identifier and current worker state give the executor a way to reconcile the ambiguous call. Where the platform supports an idempotent API, the same identified request can be repeated without multiplying its effect. That protection comes from the destination's actual API behavior; the workflow cannot create it merely by describing a retry as safe.

Multiple executors complicate the same problem. A local cooldown does not stop another executor from acting on that worker. A shared operation record or suitable concurrency mechanism must keep the total action within the service's limit, including while an ambiguous result is being investigated. Otherwise several individually bounded workflows can collectively remove too much capacity.

Some effects will outlive the restart. A message may have been delivered or a payment submitted, and another command cannot simply undo that history. The runbook should explain when to stop and what evidence the next responder needs to inspect those consequences. Uncertainty that cannot be safely resolved belongs with a named owner.

Command completion and service recovery

Once the platform reports command completion, the service still needs its own check. Falling queue age, resumed successful transactions or recovery of the affected request path can establish an outcome the command status cannot. Choose the relevant condition before the pilot so a convenient observation does not become the definition of success afterward.

Keep the initial signal, precondition checks, action request, platform response and recovery evidence in one record. A deadline determines when the workflow stops waiting, and an escalation owner gives unfinished work somewhere to go. Distinguish verified recovery, observed failure and inconclusive results; the next person needs to know which kind of problem they inherited.

Unnecessary and unsuccessful attempts are useful pilot evidence. They can reveal a faulty eligibility check, an unsuitable target limit or a recovery signal that does not establish what the team thought it did. Reviewing those cases is how a recommendation-only trial can earn a specific scope of supervised or automatic execution.

Command success still needs a service check

Read diagram description

Illustrative bounded remediation: establish current eligibility, coordinate one operation, and check an independent user signal. An ambiguous write result requires reconciliation; failed recovery requires a stop or escalation, not an uncontrolled retry loop. Diagram labels: Current eligibility: Target unhealthy, spare capacity, no conflicting action; One bounded operation: Shared operation record, target limit and cooldown; Destination result: Reconcile any timeout before another write; Independent service check: Queue age, user transactions and adverse effects; Verified recovery: Record supporting evidence; Failed or inconclusive: Stop and escalate with the record.

Two ways to read the pilot’s 24 recoveries

For the worker restart, the agreed recovery condition might require queue age to fall below an established threshold within a defined window without increasing failed jobs. The workload should determine those values. The measurement then follows all eligible incidents rather than only the repairs that looked successful.

In an illustrative pilot with 40 eligible incidents, suppose automation attempts 30 and verifies recovery in 24. Recovery among attempts is 24 / 30 = 80%. Verified automated coverage of the eligible incidents is 24 / 40 = 60%. The first describes the attempted repairs; the second also accounts for cases the automation did not attempt. Both leave important work visible: six unverified attempts and ten incidents without an attempt.

Break the unverified cases into observed failures and outcomes that could not be established. Track unnecessary actions, harmful changes, repeated incidents, manual follow-up and detection-to-verified-recovery time. Compare similar incident types and include unsuccessful attempts when interpreting duration. The SRE KPI measurement guide develops the denominator and window choices behind that comparison.

The restart settings carried into rollout

The incident runbook template can carry the decisions reached during the pilot. A useful rollout record includes these connected fields:

• Scope and evidence: the failure mode, service owner, permitted targets, exclusions and fresh signals that establish or disqualify eligibility.
• Authority: the execution identity, allowed operation and approval policy for that state.
• Execution limits: the shared operation record, maximum targets, cooldown and deadline.
• Outcome and recovery: the independent success condition, observation window and adverse effects to check.
• Stop and review: the response to timeouts, failed recovery or unavailable rollback, plus the audit record and person who can disable the workflow.

For this restart, unattended execution depends on current worker health, spare capacity and the absence of conflicting work. Target limits and cooldown constrain the repair; the recovery check and escalation route handle what follows. These settings turn approval of a familiar command into permission for a particular operation under conditions the executor can check.

A migration, concurrent change or loss of monitoring can make the previously tested restart unsuitable. When one of those conditions removes the basis for recovery, the workflow needs to return to supervised operation until the new state is understood.

Rehearsing the lost response shows what the responder will inherit from a partial restart: the operation identifier, the latest worker state and the observations still missing. A workflow that can reconcile that state and measure recovery has a basis for the next action. Otherwise, it has enough context to stop and let a person continue without repeating the request blindly.