Where should a million job IDs live?

You want to find one customer’s export, so you add its job ID to a metric. It is a helpful detail. Then the next job adds another ID, and the next adds another. The measurement system now has a growing collection of separate records to maintain, all because you wanted to look up one report.

A metric’s labels describe the measurement: its region, outcome or deployment version, for example. Measurements over time for one distinct label combination form a time series. The number of distinct combinations is called cardinality. The job ID is not useless; it simply raises a different storage question from “How many exports failed in region B?”

Start with a hypothetical export counter covering three regions, four outcomes and two deployment versions. If every combination appears, that is 3 × 4 × 2 = 24 series, before instance or other labels are added.

A million IDs do not stay one helpful detail

Add a million possible job IDs and the theoretical combination count becomes 24 million. That is an upper count of combinations, not a claim that all occur. A particular job normally belongs to one region and one outcome; actual series depend on the combinations that really appear.

The change that matters is continuing growth. Three known regions provide a bounded set of values. New jobs keep supplying new IDs, each potentially creating series and more ingestion, indexing and storage work. A field that looks small in one record can be expensive across all the records it causes to exist.

Removing region would reduce cardinality too, but at the cost of a useful question: is B failing while the others work? Keep dimensions that locate the affected population. An ID used to inspect one execution can often live in appropriately protected logs or traces instead.

That move does not erase its storage cost or sensitivity. Those records need their own access controls and retention choices. It puts individual detail where the investigation uses it without requiring every metric to carry the same detail.

A thousand traces can still miss the interesting hundred

Suppose Dispatch processes 100,000 exports and keeps traces for 1,000. Counters can still count all recorded export events; traces describe selected executions. A thousand examples may sound substantial, but they are not automatically a miniature of the full population.

Consider a specific policy: near the start of each trace, keep it with a 1% chance, independently of every other trace. That is uniform independent 1% head sampling. Now suppose a particular failure occurs 100 times. Each of those traces has a 99% chance of being left out.

For all 100 to be omitted, multiply 0.99 by itself 100 times: 0.99¹⁰⁰, or about 36.6%. Under this policy, it is entirely possible to inspect plenty of ordinary traces and find none from the failure group. The empty search does not prove the failures never happened.

Tail sampling waits for trace information before choosing what to keep, which can help retain slow or failed work. The resulting collection then favors those kinds of execution. That is useful for investigating them, but its proportions are not an unbiased estimate of the whole population without an appropriate sampling design.

Keep a route from the count to an individual job

For this Dispatch proposal, metrics keep enough dimensions to locate a region and outcome. Selected slow and failed traces retain the correlation needed to inspect individual exports. Document contents stay out of telemetry. Choose retention around the time investigations need and the organization’s requirements, rather than letting “keep everything” stand in for a decision.

Try three questions: where is the regional failure, what happened to this job, and what fraction of recorded exports failed? Name which record answers each. The fleet count explains how much work was affected; a selected trace helps explain a particular execution. Neither needs to pretend to be the other.

When dropping a field, name the question that becomes harder too. Removing per-tenant metrics may prevent a direct comparison for one tenant. An aggregate count followed by a controlled log search might be acceptable. If it is not, choose another bounded dimension or an access-controlled analysis method that meets the need.

Finish by following one failed export through the proposed design. Find its population count, then whatever individual detail was retained. If its trace was never kept, say so. The job ID still has a useful home, and the smaller dataset has an intelligible limit rather than a silence someone must mistake for success.

Sources

Prometheus label guidance and OpenTelemetry sampling explain cardinality and selection. The counts and probability example are synthetic.