What did those hundred canary exports actually test?

On our fictional release worksheet, the new worker has completed a hundred exports without an error. It looks ready for more traffic. Then someone notices that all hundred reports were small, while the changed code runs only for large exports.

The worker has been busy, but not with the work needed to answer the release question.

A canary release sends a limited share of work through a change and compares the result with the unchanged service before expanding. In this paper exercise, the Dispatch worker changes how reports are written to object storage. No live deployment is involved. The task is to choose work that can reveal what the change does, then make the decisions to expand, stop and recover clear enough to use.

Give the new path work that reaches it

Suppose only large exports use the new writing path. Define eligible work accordingly: accepted large exports in a specified region, using synthetic or authorized data for the relevant test stage. Small reports may tell you something about the worker, but they cannot demonstrate behavior they never invoke.

The unchanged workers supply the control, the group used for comparison. Give them similar work during the same period. If they receive small reports from warm storage while the new workers receive large reports after an idle period, you have changed the workload as well as the version. A slower result could have more than one explanation.

Comparing similar large exports removes some of that confusion. Keep the sizes, tenant scope and dependency conditions alongside the processing version. Then the comparison describes what each group actually did, rather than leave two appealing lines on a chart with their workloads forgotten.

Find out which worker made the file

Dispatch accepts an export before the file exists. That acceptance cannot tell you how the new writing code performed. Follow the relevant exports through completion and inspect their results.

A shared queue makes the worker identity worth checking. An export can enter through the intended canary route and later be picked up by an old worker. Its completed report is real, but the new worker did not make it. Record the processing version on the completed result, not just the route used by the incoming request.

Now revisit those hundred exports. Which reached the changed path? Which version processed them? Were the files correct and available on time? With those answers, the count begins to say something useful about the release.

Agree on what would change the decision

Watch completed reports, missed deadlines, incorrect or missing files, and the age of waiting work. Leave time for the relevant background jobs to finish. Five minutes on the clock cannot tell you the outcome of an eligible job that is still running.

One possible progression rule is to expand after the agreed sample of large exports finishes with correct content, no unexplained difference from the matched control, and a stable backlog. The service must supply evidence for the sample and thresholds. This exercise cannot provide a number that is safe for every release.

A stop rule needs the same clarity. For example, any confirmed corrupt export could halt progression immediately and page the service owner. State who can stop the rollout and who can authorize the next step. Those decisions are easier to make before a disappointing result arrives than while the team is trying to agree on what it means.

The old program may meet new files

Suppose the new worker has already written a format the old worker cannot read. Stopping further exposure limits what happens next, but reinstalling the old program leaves those files where they are. Restoring the software version has not restored the old situation.

Plan for that compatibility before relying on rollback. An expand-and-contract migration changes the system in stages, allowing old and new forms to coexist during the transition. Depending on the change, a corrective version that can handle the files already written may be another recovery route. Choose and check the route that fits the actual data.

Unfinished exports need a place in that plan, too. Which worker takes them after a stop? What prevents duplicate processing? How will affected files be checked? If those answers are still missing, reduce the first exposure or change the migration design so the work released fits the recovery available.

New successes can hide an older wait

Add this fictional result to the worksheet: new exports succeed, but the canary’s oldest waiting jobs keep getting older. Under the proposed rule, that does not justify expansion. Easy new jobs can finish while another group stays stuck. Find the versions and workloads of the older jobs before choosing the next check.

Read the completed plan once as the person deciding whether to expand and once as the person responsible for recovery. It should identify the eligible reports, comparison, observation period, stop decision, and recovery for files and unfinished work.

The opening hundred exports now have a more precise job to do. They need to answer the question about the changed path, not merely prove that a worker was active. That lets the next decision account for both the reports that finished and the work the new version would leave behind if it had to stop.

Evaluating a limited release

Canarying releases explains comparison and staged evaluation. The Dispatch conditions in this exercise are hypothetical.