Opens in a new tab
Explainer

What an empty log search can tell you

In brief

A worker restarts mid-job. To decide whether to retry, the responder needs to know what the worker recorded and what made it into search.

5 min read

Is this the right starting point?
Before you read
Basic application logging and incident-investigation concepts, including collectors, query scopes and service operation identifiers.
What you will take away
Inspect event fields, collection health and query scope to understand what an empty log search can and cannot tell you.
Sources
A magnifying glass examines an empty archive compartment while an ochre ticket rests in the adjacent compartment.
On this page3 sections

A worker restarts halfway through a job. The person on call searches for the job’s last action and gets “No matching records.” The search box has reported exactly what it knows. It cannot say whether the action never happened or its record never reached the search box.

A log records an event a program or system reports. A collector can carry that record to a place where someone can search it. When a search comes back empty, there are two paths to follow: the work the program was doing and the route its record took. In this example, the decision at the end of both paths is whether it is safe to try the action again.

Leave a clue the next responder can follow

Suppose the responder broadens the search and finds a line just before the restart: “retrying request.” It is a better clue than a blank screen, but there is no operation ID and nothing about the previous attempt. Did the destination reject it, accept it, or stop answering? Those lead to different decisions. Adding five more lines that say “retrying request” would leave the responder in the same place.

Record transitions people will need to distinguish, such as a dependency timing out, a job completing or retries running out. Give events stable names and separate fields for the details someone will search: time, environment, operation ID and what the program actually observed. Keep duration units consistent. If one component uses “status” to mean the network call succeeded and another uses it to mean the job finished, a query can run perfectly and answer the wrong question.

An operation ID gives the responder something to look up at the destination if that system supports it. A trace ID can connect the record to the rest of the request’s path. Neither needs to come with a copy of the request body. Leave out secrets and personal data where possible, and check the exception messages too. The line you did not write yourself can still carry a detail you never intended to store.

A quiet search can start with a full queue

Imagine the storage service slows down while the worker keeps writing events. If the application waits for delivery on its work path, that wait can slow the work itself. A collector can buffer records up to a limit; depending on the setup, it may reject or drop records when the queue fills. The queue, rejected and dropped record counts, and freshness of arrivals help the next person see whether the missing evidence coincides with a delivery problem.

Keep both clocks when the pipeline supports them. OpenTelemetry’s log model separates when an event happened from when the collection system observed it. An event created at 10:02 and first observed at 10:05 belongs in the worker’s 10:02 timeline. The later time helps explain why a responder looking at 10:03 may not have seen it. It is the observation time at that point in collection, and delivery to the search store can take longer still.

An empty log result has several explanations
A blank log search branches into checks of delivery and query scope before the responder interprets it.
An empty search sends you along two paths: check how the event traveled, then check what the query could see.
Read diagram description

Start with a search that returned no records. On the collection side, look for a known event and check queues, rejects, drops and arrival freshness. On the query side, check the time zone, interval, index, access and retention. These checks can expose gaps in collection or query scope. Finding one known event does not show that every event was recorded. To find out whether the operation took effect, look for separate evidence at the destination.

Find something that ought to be there

Start with the affected operation or with the user outcome that made it worth looking. Metrics can show how widespread a problem is when their population and collection support that conclusion. If you kept only a sample of the logs, a count of matching messages is a count of retained records, with no automatic claim about all the failures.

Pick a request you know was recorded or send a controlled test event. Look for it in the expected time window and time zone, in the intended index, using the access the responder actually has. Check how long the records are kept. If the known event is missing, follow it through collection and query scope before reading meaning into the worker’s silence. If it appears, you know that event was retrievable under those conditions; an older interval, a different index or a different role could still yield less.

Keep enough ordinary diagnostic context for the questions nobody has thought of yet. That may include the worker and operation IDs, event times and a clear record of what was known at each attempt. Ask the next responder to find them under their usual access. A field that only the logging administrator can retrieve on demand is a poor handoff at the moment someone else needs to make the call.

Back at the worker, the useful line tells you which operation to check. The destination can then supply evidence about its own effect, where available. If the operation ID or the collection history is missing, carry that gap into the retry decision. The empty search box has already told you as much as it can.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next