On this page4 sections
Imagine trying to free space for a Linux application that cannot write a file. You find a large directory under /tmp. The name sounds promising: temporary files should be the easy part of a cleanup.
In this hypothetical installation, that directory is where an application volume has been mounted. It holds recovery data. Deleting it would remove something the application needs, despite the reassuring name above it.
The useful first move is to follow the file the service cannot write. Its path is an address in the filesystem the process can see; a mount can attach another filesystem or storage volume at that address. Keep the exact error with the path. Together they help you find which storage the application reaches and what is stopping the write, before deciding whether cleanup belongs in the answer at all.
The directory names are still worth learning
The surprise under /tmp does not make the directory layout useless. These names give you a sensible place to look, and often tell you how long the contents are meant to live. Configuration, a service's saved state and a socket created when it starts have different jobs.
| Path | What you would conventionally look for |
|---|---|
| /etc | Settings for the host and its applications |
| /var/lib, /var/log | Application state that needs to persist, and conventional log storage |
| /run | Runtime files such as sockets and process identifiers, generally recreated after boot |
| /tmp, /var/tmp | Temporary data with different retention conventions; check the local cleanup policy |
| /usr, /opt | Installed programs and libraries, and optional application packages |
| /proc, /sys | Interfaces to the kernel and processes; writing some entries changes the running system |
| /dev, /boot | Device interfaces and files used for booting |
| /home, /root, /srv, /mnt | User homes, the root user's home, service data and conventional places to mount filesystems |
The Filesystem Hierarchy Standard describes these conventional roles. They help with moving a service as well as fixing one: settings under /etc, durable state under /var/lib and runtime files under /run have different backup and restart needs. Entries under /proc and /sys are interfaces to the running kernel, not simply another collection of stored documents.
Actual distributions and container images can arrange things differently. Many systems make /bin, /sbin and /lib symbolic links into locations under /usr: one name points to another place. Seeing two familiar names does not mean you have found two separate stores of space.
Back at our recovery directory, the conventional name suggests a short life for the files. The application's use of the mounted volume says otherwise. The directory map has helped us ask the question; the installation has to supply the answer.
Which filesystem is behind that address?
Start with the affected service path on a Linux host where these utilities are installed and you have appropriate access. Replace the example path below with that path. These commands inspect it rather than changing it:
findmnt -T /var/lib/example-service
df -h /var/lib/example-service
df -i /var/lib/example-service
ls -ld /var/lib/example-service
findmnt -T identifies the filesystem containing the path. df -h reports its space in readable sizes. df -i asks a different capacity question: are there file records, called inodes, available? ls -ld displays the directory's ownership and permission information. A filesystem with free bytes can still lack another requirement for creating the file.
A container makes “the path” a little less obvious. Its mount namespace is its view of the mounted filesystems, and its volume mapping connects paths in that view to backing storage. A directory with the same name on the host need not reach the same filesystem. Follow the service's view and its volume mapping, rather than treating the host's matching name as proof. That is how our apparently disposable directory becomes recognizable as application recovery storage.
There is another way a cleanup can disappoint. A process can keep a file open after its last directory entry has been removed. You no longer see the name, but the storage remains allocated until the last open reference closes. Deleting more files will not close that reference; it can lose more data without freeing the space tied up in the already deleted file. With the appropriate tools and access, inspect open deleted files before choosing a remedy.
If space and ordinary permissions look sufficient, examine the mount options and security policy as they apply to the service's own account. An administrator being able to write is not the same test. A read-only mount or a policy restriction may be doing exactly what it was intended to do. Broadening access before finding the difference can remove that protection without explaining the failure.
You do not need a complete map of the machine to narrow these possibilities. You do need to connect this path to its mount, the application's account and the lifetime of the data. Before any destructive cleanup of the recovery volume, establish its retention and recovery requirements and who is responsible for the change.
Trace the checks in order
This proposed investigation moves from the exact failure to the storage and access rules involved. The final step is a separately owned intervention, not an action performed by the preceding read-only commands. Diagram labels: Failing path + service identity: Keep the exact error and environment; Namespace + mount: Resolve container volume and underlying filesystem; Capacity + inode availability: Inspect the filesystem containing this path; Permissions + mount policy: Check ownership, read-only state and security policy; Owned intervention: Preserve evidence before changing the boundary.
Let the failed write tell you where to look
At this point, “the disk looks fine” is no longer the only observation. You know which filesystem matters and can compare its condition with the application's actual error. A capacity problem, a read-only mount and a denied operation lead to different investigations.
Find the error where the service is configured to record it. There may be no /var/log/syslog on this system. A systemd service may use the journal, its collected event log, while a container may send output to a separate log collector. Check the logging configuration rather than treating an absent familiar log file as silence from the application.
On a systemd-based system, with appropriate access, substitute the affected service name in these checks:
journalctl -u example.service --since "30 minutes ago" --no-pager
systemctl status example.service --no-pager
The journal query limits its entries to the recent thirty-minute window for that service. The status command shows the service's current status. Keeping the log query focused on the failure also avoids unnecessarily collecting sensitive data. Save the relevant error and mount options before changing them, so the next person can still see what blocked the write.
Choose the change that fits this write failure
If capacity is being consumed, find out how the data is meant to grow and expire before selecting files to remove. If access was denied, compare the service's account with the access it is supposed to have. Resource pressure without exhausted capacity belongs in Linux performance diagnosis; deleting files may not touch the resource that is limiting the work.
The same connection matters for restarts, mount changes and signals sent to processes. A forced kill can discard work held in memory and skip cleanup. Finding a process with an open deleted file explains where storage is held, but does not establish that killing it is an acceptable way to get the storage back. The responsible team and recovery procedure belong in that decision.
Our hypothetical cleanup now has a different starting point. The directory under /tmp is application recovery storage, with a particular backing filesystem and a failed write to understand. It may need a capacity change, an access correction or a retention decision. The familiar name got us looking in the right area; following the path kept us from mistaking recovery data for rubbish.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- Filesystem Hierarchy Standardrefspecs.linuxfoundation.org
