On this page4 sections
The canary has run for an hour, and the error chart is quiet. In this hypothetical release, the team is about to send it more traffic when someone checks the request mix. The change affects writes. The new version has served only reads. The team limited how many customers could encounter a problem, but it still has no evidence about what happens when they use the changed operation.
A canary gives a candidate version a bounded share of production work while the existing version provides a baseline. Before raising that share, ask what the candidate actually handled, how both versions behaved and what it would take to withdraw the candidate.
Follow a payment through the path that changed
Suppose a customer submits a payment to the candidate. Did the transaction complete correctly? How long did it take? Did the work queued after the response finish too? Count eligible writes and measure those outcomes on both versions. A single good payment among thousands of reads may tell you that the path can work, with little evidence about how often it will.
Assignment determines what you can compare. Keeping users on one version may preserve a session across requests. Splitting individual requests can send the same session through both. If the candidate serves only a handful of tenants, check whether their traffic differs from the baseline’s before attributing a slower write to the release. Shared state can also let one version affect the other. Google’s canarying chapter explains how population, duration, workload and metrics shape what a comparison can show.
Check how traffic actually reaches the two versions. One candidate in a hypothetical ten-replica deployment does not establish that it handled a tenth of the requests or writes. A Kubernetes rolling update controls how replicas change. If you need a particular request weight or automated analysis, check which routing and rollout tools supply it and compare their configuration with observed work.
The payment can outlast the candidate
Suppose the first batch of writes reaches the candidate and the team needs to stop. Keep the baseline version and enough capacity to take back its traffic. Then ask what the candidate has already left behind. The baseline must still be able to use the current schema. A published message still has a consumer to reach. Redirecting a request does not reverse a payment, restore deleted data or undo a migration.
Test withdrawal and any separate state recovery in a safe environment before you rely on them. Completing a rollout stage tells you where deployment got to. To decide whether to expand, the team still needs the write outcomes and evidence that it can stop or contain the change.
Two slow versions can agree with each other
Now suppose writes reach both versions, and a shared database slows them equally. The candidate looks as fast as the baseline, but neither is meeting the service level objective, or SLO. Keep a check against the service’s own target beside the comparison. If an unrelated incident changes the workload or spoils the baseline, pause and keep the observations without treating the old comparison as decisive.
Read diagram description
Send bounded representative traffic to the candidate and baseline, with recovery capacity available. Compare user outcomes, counts and workload mix, and check whether the service meets its target. Promote in steps when the evidence supports it, hold when evidence is insufficient, and stop or mitigate when a failure condition is met.
Decide before rollout what each result means. Promote in steps when enough representative writes complete within the agreed limits and withdrawal remains available. Hold when too few writes arrived, a relevant scheduled job has not run or the evidence cannot distinguish the candidate from another cause. Stop or mitigate when a predefined failure condition trips. Write down the changed behavior, minimum evidence, observation window and stop criteria together. An hour that passed without a write was time on the clock. It was no evidence of write reliability.
Keep the limits with the decision
A representative canary still may not reproduce a rare customer configuration, a memory leak that takes days to emerge or a problem that needs full traffic. Use separate tests where those risks matter, and keep monitoring and recovery available as you expand. The release gate guide connects the decision to policy. The AI release engineering article explains why a suggestion from a tool still needs independent checks.
At the end of the hypothetical hour, give the next decision maker something more useful than “canary green.” Tell them which writes ran, what the customers and the queued work saw, and what would remain if the candidate went away. If the answer is that no eligible writes ran, the decision is to hold until the release has something to show.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- canarying chaptersre.google
