Opens in a new tab
Guide

A runbook that helps the next responder choose

In brief

Use a downloadable SRE runbook template and a database-connection example to connect observations, permitted actions and recovery checks.

7 min read

Is this the right starting point?
Before you read
Basic service troubleshooting, database connections and deployment concepts; local access and approved procedures are required before applying the template.
What you will take away
Adapt a runbook into diagnostic branches with bounded actions, compatibility checks, independent recovery and a usable escalation route.
Sources
A branching procedure leaves an action lever unpulled beside a closed evidence shutter and carries observations to an open handover tray.
On this page5 sections

The responder has found the runbook. It says to restart the application. The service is also halfway through a database migration, and the person who wrote the procedure is asleep. Now the useful question is whether that restart is still appropriate.

A good runbook helps a qualified responder answer that question and keep going when the answer is unavailable. It connects the symptom to checks, the checks to permitted actions, and the actions to evidence that customers can use the service again.

Download the Markdown runbook template for that structure. Add the service's actual details, check its access requirements and rehearse it before use. The hypothetical database-connection example below shows how the fields become a working investigation. The template itself grants no permission to run commands in an unfamiliar environment.

Leave the reasoning beside the action

A useful procedure lets its reader follow the same questions the author would ask. Does the symptom fit? Where is the request waiting? Which action changes what was found, and what would show that it worked? Put the answers where the responder needs them.

SectionWhat the responder needs
Scope and ownershipService, environment, related alert, owner, required access, and last tested date.
Confirm the symptomUser impact, exact dashboard or query, time window, expected result, and signs this is a different failure.
DiagnoseOrdered checks with branches that explain what each result means.
ActPreconditions, approved command or workflow, target limits, expected effect, and stop conditions.
Verify and escalateIndependent recovery checks, observation window, reachable owner, and evidence to hand over.

These fields give a restart its context. The reader can see the environment it applies to, what must be true first and when to stop. Without that reasoning, a familiar command can become the answer to several very different failures.

Names can help people find the document too. Some teams call a broad coordination guide a playbook and a specific operating procedure a runbook. Agree on the convention your team uses. The distinction is useful when it helps the reader reach the right material.

Find which side of the connection is waiting

Return to our hypothetical migration. Requests are waiting for database connections, and restart compatibility is unresolved. We have two questions to pursue together. Where are requests getting stuck? Which changes remain permitted while the migration is running?

The application keeps a pool of connections ready for database work. A request borrows one, uses it and returns it for another request. If every usable connection is occupied, the next request waits at the application. The database may still have spare capacity even though the application cannot hand out another connection.

Compare the pool's occupancy and wait time with the database's connection count, active queries and resource pressure. A full application pool and a saturated database can both leave requests waiting. They point to different investigations and different capacity decisions.

For the full pool, look at what is holding the connections. Long transactions and legitimate load can keep them busy, just as a leak can. Recent code, configuration and instance-count changes help explain why the demand or holding time changed. An occupied connection alone does not identify the cause.

Now imagine adding more application replicas to a database that is already under pressure. Each replica may open its own pool. The application gains more ways to ask for work while the database receives still more requests. Raising the database connection limit also allocates resources and may leave the limiting query or resource unchanged.

The arithmetic makes this visible. With one independent pool connected directly to the database per application instance, a maximum of twenty connections permits up to eighty across four instances and 160 across eight. Those are configured upper bounds, not observed usage. Other clients and reserved database slots still count. HikariCP's pool-size reference explains the local limit, and PostgreSQL's connection settings explain the server-wide constraint.

A proxy can change that relationship. With transaction pooling, a backend connection returns to the proxy's pool when the transaction ends. Client connections need not each hold a backend session throughout their lifetime. Include the proxy's backend counts before estimating what more replicas would require from the database.

Pool exhaustion is not proof of database saturation
Pool exhaustion is not proof of database saturation. Illustrative diagnosis: compare application pool waits with database state before choosing a mitigation. Adding application replicas or connections can worsen a saturated database. Missing evidence needs an authorized fallback.
Illustrative diagnosis: compare application pool waits with database state before choosing a mitigation. Adding application replicas or connections can worsen a saturated database. Missing evidence needs an authorized fallback.
Read diagram description

Illustrative diagnosis: compare application pool waits with database state before choosing a mitigation. Adding application replicas or connections can worsen a saturated database. Missing evidence needs an authorized fallback. Diagram labels: Connection-acquisition waits rise: Confirm customer impact and the affected application; Database has capacity: Investigate held connections and transaction behavior; Database is saturated: Investigate demand, queries and resource pressure; Bounded, evidence-based mitigation: Check compatibility; verify waits, transactions and DB health.

Choose the change the observations support

A rollout just before the waits began gives the responder a useful lead. Check its approved rollback path and compatibility with the current data before choosing it. If excessive demand is the immediate pressure, an already tested traffic or concurrency limit may help. If a particular query needs cancellation, involve the database owner in checking transaction and retry consequences. Cancelling a query and closing its session do different things.

For the chosen action, record the target and current state, the expected effect and the condition for stopping. Observe its effect before adding another change. If the cause is still unclear, the next responder should receive the pool waits, database pressure, rollout timing and attempted actions that have narrowed the question.

The action record also separates asking from observing. In our example, write down which instance was targeted, which approved step permitted the change and when the executor sent it. Keep the tool's acknowledgement beside separate checks of pool waits, customer transactions and database pressure.

If that acknowledgement is lost, the outcome is unknown. Inspect the target before repeating a change that may already have completed. The decision-log replay kit includes a synthetic missing-result branch for practising this part of the handoff.

Take away the dashboard during rehearsal as well. Can the responder reach another authorized source or the person who can decide? Losing a prerequisite check changes the available path. It does not turn a conditional restart into permission to try one and see.

An unfamiliar failure may fit none of the listed branches. A useful runbook still gives the responder an escalation point and a record of what has been observed. It leaves room for investigation instead of steering every symptom toward the same command.

Watch the transactions return

The pool waits fall. That is encouraging, but requests might also have stopped arriving. Check that traffic continues and customer transactions succeed while database pressure remains acceptable through the agreed observation window. Those signals together distinguish a relieved bottleneck from an application that has simply gone quiet.

Keep unresolved risks attached to their owners. Repair any link, permission, branch or expected result that the incident exposed as wrong. Record the document's edit date separately from the date its procedure was exercised. A corrected link and a rehearsed recovery each tell the next responder something useful.

Hand the procedure to another responder

Ask someone other than the author to use a safe exercise or representative staging setup with the access available on a real shift. Can they identify the next permitted action, its stopping condition and a fallback when evidence is missing? Wherever the exercise needs an unwritten explanation, the document or its supporting access has more work to do.

AI can help arrange the draft. People who know the environment still need to establish whether its commands and assumptions fit. Once the preconditions and recovery checks can be executed and tested, the procedure becomes a firmer basis for automated remediation.

Our responder may still be waiting for the migration's compatibility answer. They now know what evidence is missing, whom to reach and what to hand over. That ability to continue is what the runbook's author can leave behind, even when the familiar restart is unavailable.

Put this into practice

Draft a procedure another responder can use

Bring: A specific failure scenario, service scope, access requirements, escalation contacts and recovery checks.

Source context

Source links appear within this article. Read them alongside the author’s analysis and evaluate the guidance against your environment.

Report an error or outdated detail

Keep reading

Where to go next