The queue is draining. How old are the errors?

The error graph lights up just as the monitoring backend returns. A responder reaches for the restart control, then pauses. Is the service failing again, or have four minutes of old errors finally arrived? The graph is available now. That does not mean the trouble it describes is happening now.

Imagine a hypothetical inference service continuing to produce spans, records of individual operations, throughout a four-minute telemetry outage. A queue can hold those records for later without making them any younger. An assistant deciding whether to intervene needs to know both whether they arrived and whether they are recent enough for that action. The monitoring-freshness principle makes room for a state that matters here: current visibility is unavailable.

Find where the missing span could be waiting

Start with one span leaving the application. It may pass through a local OpenTelemetry Collector, which receives and forwards telemetry, then a gateway before reaching the backend that indexes it for readers. Mark those handoffs. If the application's instrumentation library drops the span, a gateway queue cannot recover something it never received. Likewise, a processor's private memory does not become durable because an exporter later writes to disk.

The OpenTelemetry resilience documentation describes several protections with different jobs. A sending queue holds work awaiting export. Retry settings control further attempts after retryable failures. Persistent storage can retain queued work across a Collector restart. Choosing one does not make the entire journey durable.

If the decision cannot wait, use another observation path or suspend the action. A small synthetic request, made for testing rather than by a customer, can check a customer-facing path while traces are unavailable. Its success describes that request. It neither fills in other dependencies nor explains what the missing traces would have shown.

New arrivals still need a place while the backlog drains

Before calculating space, ask what the queue counts. The exporter-helper configuration supports sizing by requests, individual items or serialized bytes, meaning the encoded data's size. Check support in the exact exporter and version you will use. One request can contain many spans; counting requests as though they were spans can make a carefully planned queue much too small.

Suppose records arrive steadily at λ items per second and delivery can sustain μ items per second. Use the same item unit for both. A complete delivery outage lasting T seconds adds λ × T items to the backlog, B. When delivery returns, new arrivals still take λ of the available capacity. The backlog therefore has μ − λ items per second left to use, giving an estimated catch-up time of B ÷ (μ − λ), and only when μ exceeds λ. This accounts for capacity, not scheduling or memory consumption.

Put numbers on our four-minute example. At 1,500 spans per second, with none delivered or lost, 360,000 spans accumulate. If the destination then accepts 2,000 per second, only 500 per second can clear the old work: roughly 720 seconds, or twelve minutes. Accepting 1,500 merely keeps pace with arrivals. More disk can lengthen the wait the queue survives; it cannot make that backend catch up faster.

The twelve-minute estimate still does not tell you how old each arriving span will be. Retries may let newer work overtake older work. Measure age from the source event at the point where the assistant reads it, and retain the event window alongside the observation. A query run at 10:05 can describe a failure at 10:01.

Question for the trial	What to measure	What still needs an answer
Will the queue hold the planned interruption?	Arrival rate, sizing unit, capacity and behavior at overflow	Whether the retained data will be timely enough for the action
Can delivery clear the backlog?	Sustained delivery capacity beyond continuing arrivals	Whether every accepted event has become searchable in the backend
Can this decision use these observations?	Source-event age and expected coverage when queried	Whether an unrelated service dependency is healthy

Set up a queue you can safely interrupt

Use a nonproduction Collector and a controlled destination. Record the distribution, version and components, a writable test volume, access to internal metrics, the allowed input rate and disk budget. Get permission to interrupt that destination and name the person who can stop the experiment. Shared production storage is not the place to discover what happens when this queue fills.

The fragment below connects storage, capacity, consumers and retries; it is not a complete or version-pinned deployment. Add it to a working test pipeline with the required receivers, processors, authentication and TLS, the encryption protecting the connection. Validate it using the actual binary intended for the trial.

extensions:
  file_storage/pilot:
    directory: /var/lib/otelcol/pilot-queue

exporters:
  otlp/pilot:
    endpoint: telemetry-test.example.com:4317
    sending_queue:
      enabled: true
      storage: file_storage/pilot
      sizer: items
      queue_size: 500000
      num_consumers: 4
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 10m

service:
  extensions: [file_storage/pilot]
  # Retain your test pipeline and reference otlp/pilot there.

These values are trial inputs. In particular, max_elapsed_time: 10m is not a promise to retain each event for ten minutes after its timestamp. Work may wait before export is attempted, and work rejected before enqueueing never enters this retry policy. Count failure to join the queue separately from a later failure to export.

The file-storage extension requires a writable directory. In Kubernetes, choose a volume that survives the particular replacement you plan to test, with appropriate separate storage for each Collector. Do not have several processes improvise around one shared writable directory. After replacement, inspect volume attachment, permissions and the retained records. A familiar path in the config is not proof that the earlier contents survived.

Send synthetic payloads without credentials or customer prompts. The volume still needs the required access controls, encryption and retention rules. Include disk use, input/output work, outgoing traffic and backend ingestion in the replay's cost. Retaining more records also retains more contents, and delivery can arrive in a burst when the destination returns.

Match the arrivals to the records you sent

Use Collector internal telemetry to follow queue size and capacity, enqueue failures, receiver refusals, export failures and sent items. Check names, labels and units against the pinned build rather than assuming every version presents them identically. A failed export can be retried, so its failure count is not automatically a count of permanently lost records.

Give a small synthetic sequence identifiers and source timestamps, and save those at the producer too. After a defined wait, compare the destination's unique IDs, duplicates, delays and omissions with that list. You can establish what happened to the sample without claiming exactly-once delivery. Keep its IDs in payloads rather than adding unbounded metric labels.

Keep watching memory while the disk queue grows. Receivers, processors and active requests still use RAM. The memory limiter can refuse work before persistent storage ever sees it. Otherwise an apparently reassuring amount of free disk may distract the team from an earlier stage already losing observations.

Watch the assistant lose its current view

Before causing an interruption, agree on two outcomes: the intended history survives the planned outage, and the assistant visibly abstains once its observations exceed the owner's age limit. Set that limit according to the consequences of acting on old information. Available disk space cannot decide how stale a restart decision may be.

1. Send the synthetic sequence at the agreed rate. Confirm delivery and check that event age stays within the decision limit.
2. Interrupt only the test destination. Watch the queue grow and exports fail, and note when the assistant's observations become too old. It should report unavailable evidence, not a healthy service.
3. While that destination remains unavailable, restart the Collector. Check recovery from the intended storage and record any losses that happened earlier in the path separately.
4. Restore delivery while new records continue to arrive. Measure catch-up and event age at query time, including delayed or throttled current work.
5. In a separate bounded test, exceed queue capacity or deny writes to the disposable volume. Verify that a working notification route reaches its owner. Do not fill shared storage.
6. Compare the final sequence with the producer's record and the decisions taken during the interruption. Account for gaps and duplicates even if queue depth has fallen.

A record arriving late may explain the outage without having been available to the responder during it. A fresh synthetic success may show present behavior while older gaps remain unresolved. Preserve both observations. Give the unexplained gaps an owner where they matter for diagnosis or audit.

Bring back the history and keep the decision honest

Begin with one owned workload. Expand only when storage use, refusals, catch-up and event age meet the agreed conditions. Retain the previous configuration and a stop procedure. If rollback would remove storage or change queue identity, deliberately stop or redirect input and account for the waiting work first. Deleting the volume would erase records the pilot still needs.

If an action needs observations younger than the outage you plan to buffer, persistence cannot meet that deadline. Keep the history, but supply another observation path for the urgent decision or wait. An hourly trend with modest consequences may tolerate the delay perfectly well. The appropriate age comes from the use, not the queue.

When that error graph lights up again, the responder should be able to see which window the errors belong to. Put the allowed event age and alternate observation path in the runbook. Then “the queue is draining” remains useful news without becoming an instruction to restart a service on the strength of yesterday's trouble.