Make the first on-call shift workable

The alert’s dashboard link works. It opens a login screen for an account the new responder does not have.

In this fictional first shift, the investigation has stopped before it reaches a single graph. A page is an alert asking someone to respond. A runbook is the written route from that symptom to useful observations and permitted action. For the person holding the page, a correct command is not much help until there is a way to reach the place where it can be used.

We can rehearse that route with a delayed export in the fictional Dispatch service, without production access. Work with a partner if possible. On your own, use the supplied complications and describe the result as a desk check.

The sample page says: Dispatch production, region A: exports accepted more than ten minutes ago are still unavailable for download. Ten minutes is an invented target for this exercise. The API can accept a request before its file exists, so a successful acceptance response cannot settle the alert. We need to follow the same export identifiers to their completed reports.

These three paper cards stand in for dashboard and log results. Their times, identifiers and observations are all invented. Name them A, B and C in the runbook so another person can find exactly the result you mean.

Paper card	Supplied observation at 09:12	What it contributes
A · export record	Export X17 was accepted at 09:00; no completion or download link is recorded.	This export has passed the exercise’s ten-minute target.
B · worker record	The assigned worker reported a heartbeat at 09:11 and still lists X17 as pending.	The worker is present; a heartbeat does not establish completed exports.
C · storage attempt	The worker’s 09:11 attempt to write X17 returned permission denied.	The next branch investigates storage access with its owner; this evidence alone does not justify a worker restart.

When the runbook sends your partner to an observation, they find the named card. If you remove the dashboard from the exercise, card A remains another way to see the export’s state. If the storage owner is unavailable, follow the backup contact route you wrote down. An empty contact field is something to repair, not an invitation to invent a helpful person. A real shift will need approved queries and tested access where these cards now sit.

Give the first decision somewhere to start

Begin with the missing exports in region A. Say where to find those same exports and the same time interval, and which account role can open the information. Then show how the results change the next step: stopped workers suggest inspecting processes, while failed storage requests point toward the storage owner.

Consider what the instruction “restart it” leaves for a new responder to work out. Which worker is “it”? What observation suggests restarting will help? Who can approve the change, and what result would show it worked? Put those details beside the proposed action, where the responder will need them.

For optional local practice, create dispatch-lab.service in Services and logs. This exercise uses a simpler symptom: nothing listens on loopback port 8080. Read the journal, then allow one start of that local unit and check both HTTP 200 and the expected page contents. Paper participants can review the proposal without executing it. That permission applies only to the lab service, not a production worker.

Read the page as the next responder

Receive the sample alert as though you had not written the instructions. Confirm the service and environment, find the card or query for that time range, and compare the affected export with the promised result. Use what you find to choose the next action.

Pay attention to where progress stops. A link into the wrong region loses the export. A missing account role prevents a query. A contact without a backup leaves everything waiting on one person. Those failures need different repairs; a stronger instruction to follow the runbook would leave all three in place.

Introduce one complication at a time. Take away the dashboard while leaving logs available, and watch how the responder finds another route to the export. The question is whether the document helps them continue, not whether they can guess what its author meant.

What happens when the right person is unavailable?

Write down primary coverage, backup coverage, specialist help and who can approve a risky change. Link to the maintained assignments rather than leave a copied name that quietly keeps last month’s rota alive.

Try three separate interruptions: the dashboard login fails, the storage owner does not answer, and two incidents arrive together. Follow each through to an alternate source, a backup specialist or someone who can allocate help. Include the agreed wait before escalating. A route that ends at a blank space has shown you a gap before the real shift begins.

Coverage also has to survive meals, handoffs and the need for relief after a long incident. Check who takes over during absence or overload. Filling the ordinary shifts on a timetable does not necessarily cover those moments.

Hand it over without the spoken tour

Give the runbook to a partner and let them use it without your explanation alongside it. Ask them to find the affected export, reach the observations, choose the first permitted action and explain when to seek help. Record what actually happens. A solo desk check is useful, but it is not evidence that another responder could follow the document.

Repair the particular link, missing permission or unexplained step that stopped them, then repeat that part. The second attempt can show whether the route became usable. A newer review date cannot show that by itself.

Finish with a route the shift can use

Keep the runbook with its access requirements, escalation route, relief arrangements and observed gaps. Before a real shift, verify actual permissions and response expectations. Paper can test the reasoning; it cannot supply an account or grant access.

Return to the login screen at the start. The responder now needs either a working route to the information or a named person who can restore it, followed by an action they understand and are allowed to take. Finding those missing connections in a rehearsal leaves time to repair them before someone is waiting for an export.

Preparing for the person on call

Google’s on-call chapter discusses preparation and sustainable response. The Dispatch rehearsal is an illustrative teaching exercise.