SRE runbook template with a database connection example

At two in the morning, the runbook’s author may be asleep. The next qualified responder must recognize the symptom, find the right check and know what its result permits. A useful template leaves that reasoning on the page: diagnosis, action, recovery and escalation.

The familiar restart can stop being appropriate during a database migration. The runbook needs to help its reader discover that condition and continue when the compatibility answer is unavailable. A list of commands cannot make that decision for them.

Download the Markdown runbook template to structure a service-specific procedure. It needs local details, access checks and rehearsal before use; the template alone does not authorize commands in an unknown environment. The worked example below shows how to connect the fields into an investigation rather than a list of plausible actions.

The parts of a usable procedure

Read the sections as a conversation with the failure. Does this symptom fit? Where is the request waiting? Which change addresses what was found? What would recovery look like? Each answer prepares the next; beginning with a restart asks a tired reader to supply all of that reasoning.

Local terminology can help readers find the right document. Some teams use “playbook” for broader incident coordination and “runbook” for a specific operational procedure. Agree on that convention locally rather than treating the words as a universal distinction.

Is the application pool full, or is the database saturated?

In a hypothetical database-connection incident, the runbook recommends a restart, but the qualified responder cannot determine whether a migration forbids it. The next checks need to answer two separate questions: where are requests waiting, and which changes remain safe while the migration is in progress? The first directs the diagnosis; the second limits the available mitigation.

A connection pool keeps a set of database connections available for application work, so a request can borrow a connection rather than establish a new one for every operation. If all usable connections are occupied, another request may wait for one to be returned. That wait happens at the application even when the database itself has spare capacity. The distinction is the starting point for the checks below.

Begin by comparing application pool occupancy and wait time with database connection counts, active queries and resource pressure. The application pool can exhaust its available connections while the database still has capacity. Conversely, several individually healthy-looking pools can together demand more connections than the database can support. Both situations can produce waiting requests while calling for different changes.

A full pool is not proof of a leak. Long transactions, legitimate load or more application instances can occupy the connections too. Compare recent code and configuration changes, transaction duration and instance count to find what is holding the pool and what has changed in demand.

Adding capacity on the wrong side can intensify the pressure. Extra application replicas can open more connections against an already saturated database. Raising the connection limit can consume additional resources without improving useful throughput. The pool and database observations therefore matter to the choice of capacity change, not just to describing the symptom.

A sizing check makes that interaction concrete. In an illustrative configuration with one independent, direct-to-database pool per application instance, a twenty-connection maximum permits up to eighty connections across four instances and 160 across eight. These are configured upper bounds, not observed usage; other clients and reserved database slots still matter. HikariCP’s pool-size reference explains the local limit, while PostgreSQL’s connection settings describe the server-wide constraint.

If a proxy uses transaction pooling, client connections need not map one-to-one to backend sessions. The runbook then needs the proxy’s backend counts as well as the application’s pool counts before a replica increase can be assessed.

Pool exhaustion is not proof of database saturation

Read diagram description

Illustrative diagnosis: compare application pool waits with database state before choosing a mitigation. Adding application replicas or connections can worsen a saturated database. Missing evidence needs an authorized fallback. Diagram labels: Connection-acquisition waits rise: Confirm customer impact and the affected application; Database has capacity: Investigate held connections and transaction behavior; Database is saturated: Investigate demand, queries and resource pressure; Bounded, evidence-based mitigation: Check compatibility; verify waits, transactions and DB health.

How the observations change the mitigation

If the behavior began with a rollout, inspect its approved rollback path and data compatibility. If excessive demand is the immediate pressure, an already tested traffic or concurrency limit may be appropriate. If a particular query needs cancellation, the database owner should help establish its transaction and retry consequences. Killing a query and closing a session are different operations.

After choosing a path, record the target and its current state, expected effect and stopping condition. Observe that change before piling on another. If the cause remains uncertain, hand the next responder the pool waits, database pressure, rollout timing and attempted actions that narrowed the question.

In the hypothetical pool incident, an action entry can say which application instance was changed, which approved step permitted it and when the executor sent the request. Keep the tool’s acknowledgement separate from falling pool waits, resumed customer transactions and database pressure during the observation window. If the acknowledgement is lost, record the action outcome as unknown and inspect the target before repeating a potentially completed change. The decision-log replay kit gives another responder a synthetic missing-result branch to practice alongside the runbook.

Now make the dashboard unavailable in rehearsal. Can the reader find another authorized source of evidence or the person who can decide? Losing the check must not silently turn “restart if these conditions hold” into “restart and see.”

An unfamiliar failure may fall outside these branches. The runbook can still help by identifying the point for escalation and the observations collected so far, leaving the responder free to investigate rather than forcing the incident into the restart procedure.

Recovery includes successful customer transactions

Watch what happens to both the application and its customers. Falling connection waits are encouraging only if requests are still arriving and transactions resume successfully; waits can also disappear because the application stopped serving traffic. Compare those signals with database pressure through the agreed window. Together they distinguish a relieved bottleneck from an empty system that only looks quiet.

Retain unresolved risks with their owners, and revise any link, permission, branch or expected result that proved wrong. Record when the document was edited and when its procedure was last exercised as separate dates. Correcting a dashboard link keeps the document useful; rehearsing the recovery path establishes whether another responder can actually follow it.

A rehearsal with another responder

Have someone other than the author use the procedure to identify the next permitted action, its stop condition and the fallback when required evidence is missing. A safe exercise or representative staging setup should give them the access available on a real shift. Any point that requires unwritten knowledge exposes work in the procedure or its supporting access.

AI can help arrange a draft, but environment-specific review still has to establish whether its commands and assumptions are suitable. The procedure becomes a stronger basis for automated remediation only once the preconditions and recovery checks can be executed and tested.

The next responder can now follow pool waits and database pressure to a fitting action. If the migration still leaves compatibility unknown, they have a person to reach and evidence to bring. That is what the author leaves behind: a way to continue the reasoning, even when the familiar command is unavailable.