Finding a Linux write failure through paths and mounts

In a hypothetical cleanup, /tmp looks like an obvious place to recover space. But the application’s recovery data is on a volume mounted there. The familiar directory name is a starting clue, not a promise that the contents are disposable.

A path is an address in the filesystem a process can see. A mount can connect that address to another filesystem or volume. Follow the exact failing path and error to discover what storage and permissions the service actually encounters before choosing a cleanup.

What the directory hierarchy tells you

The Filesystem Hierarchy Standard documents these conventional roles. Configuration, persistent state and runtime data have different lifecycles, while /proc and /sys expose interfaces to the running kernel rather than ordinary stored documents. The table helps identify what kind of data or interface deserves closer inspection. These conventions also help when moving a service: configuration under /etc, durable state under /var/lib and runtime files under /run have different backup and restart needs. A filesystem layout is therefore a useful map of lifecycles as well as a starting point for finding a write failure.

Actual distributions and container images may arrange things differently. Many systems merge /bin, /sbin and /lib into locations under /usr through symbolic links. Two familiar names therefore do not necessarily offer two separate places to reclaim capacity.

The recovery-volume example shows why the map and the installation both matter. A conventional name suggests temporary data; the mounted application’s actual use gives it a different lifetime. Following the mount reveals that difference before the data disappears.

The filesystem the service actually sees

On a Linux host with these utilities installed, substitute the affected service path in these read-only checks:

findmnt -T /var/lib/example-service
df -h /var/lib/example-service
df -i /var/lib/example-service
ls -ld /var/lib/example-service

These commands answer different parts of “why can’t it write?” findmnt identifies the filesystem behind the path. df -h inspects its space, while df -i inspects the file records it has left. ls shows ownership and permissions. Free bytes are useful evidence, but they cannot tell you that every other requirement for creating a file is satisfied.

A container adds another view to inspect. Its filesystem namespace determines the paths it can see, and its volume mapping connects those paths to backing storage. A matching name on the host does not establish a matching filesystem. This is the point in the hypothetical cleanup where the apparently disposable directory becomes identifiable as an application volume.

A process can keep a deleted file open. The name has disappeared from the directory, but its storage remains allocated until the handle closes. If that explains the missing space, deleting more files sacrifices more data without releasing the storage you sought. Inspect the open deleted files with appropriate tools and access first.

If space and ordinary permissions look sufficient, inspect mount options and security policy under the service’s own identity. An administrator writing successfully does not show that the application may write. Broadening access before finding that difference can undo an intentional protection.

Read-only checks of the path, mount, namespace and service identity can narrow the write failure without a complete system map. Before destructive cleanup of the recovery volume, connect the full path to the application's retention and recovery requirements and establish the responsible change owner. A temporary-looking directory name supplies neither that ownership nor evidence that its contents can be discarded.

Trace a write failure to the boundary that denied it

Read diagram description

Read-only investigation sequence: the same-looking path may resolve to different storage inside a container. A successful administrator write does not establish that the service identity can write. Diagram labels: Failing path + service identity: Keep the exact error and environment; Namespace + mount: Resolve container volume and underlying filesystem; Capacity + inode availability: Inspect the filesystem containing this path; Permissions + mount policy: Check ownership, read-only state and security policy; Owned intervention: Preserve evidence before changing the boundary.

What the write error adds to the diagnosis

The filesystem checks narrow the possibilities; the application's own error helps choose among them. Inspect its logging configuration instead of assuming a distribution-specific file such as /var/log/syslog exists. A systemd service may use the journal, while a container may write to standard output for collection elsewhere.

In a systemd-based environment, with appropriate access, inspect the affected service with:

journalctl -u example.service --since "30 minutes ago" --no-pager
systemctl status example.service --no-pager

The bounded time range keeps attention on the failure and avoids unnecessarily collecting sensitive log data. Preserve the error and relevant mount options before changing either, so the investigation retains evidence of whether capacity, a read-only mount or a denied operation blocked the write.

Capacity, permissions and recovery need different interventions

Capacity growth leads to questions about lifecycle and retention before cleanup. A permissions failure leads to the service identity and intended access. Resource pressure without capacity exhaustion belongs in Linux performance diagnosis, where file deletion may not address the limiting condition at all.

Restarts, mount changes and termination signals also need a connection to the diagnosis. A forced kill can discard in-memory work and bypass cleanup, so a process holding storage is not by itself a reason to kill it. Establish the responsible team and recovery procedure before choosing that intervention.

The recovery directory now has a description more useful than “large” or “temporary”: its backing filesystem, the failed write and the data the application needs. Those facts lead to different choices about retention, permissions or capacity. The name helped you start; the actual storage path tells you which choice fits.