Linux storage: why deleting a large file may not make room

Imagine deleting a large file to make room, then trying the failed write again. The same “disk full” message comes back. It feels as though Linux has not noticed the cleanup.

It may be counting something different from what you expected. The removed file might belong to another filesystem, or a running program might still have it open. Even before deletion, a file’s displayed length may differ from the storage it occupies. Two small lab files will let us see that last difference without filling a disk.

Work in the disposable Ubuntu VM and keep your real machine’s files out of the experiment. The exercise writes a ten-mebibyte file, not enough data to deliberately fill the guest. Begin by looking at the practice directory:

cd ~/linux-basics
findmnt -T .
df -h .
df -i .
du -sh .

The dot means “here,” tying each result to this directory. That detail matters on a machine where home folders, logs and application data live on different filesystems. A writer needs room on the filesystem holding its destination; spare room somewhere else cannot do that job.

Which kind of room has run out?

findmnt -T identifies the filesystem containing the path. df -h reports its space, while df -i reports its inodes, the records that hold metadata for filesystem objects. A crowd of tiny files can use up those records even while there is plenty of room left for file contents.

du takes a different route. It walks the files it can reach below the directory and counts their allocated space. That view is narrower than df’s whole filesystem. Other directories, inaccessible files, filesystem metadata and files still open after deletion can all contribute to a difference between the totals. Before trying to make the numbers agree, notice what each command was able to count.

Two ten-mebibyte files, different amounts of storage

Create one file with unwritten space and another by writing zeros:

truncate -s 10M sparse-lab.bin
ls -lh sparse-lab.bin
du -h sparse-lab.bin
dd if=/dev/zero of=written-lab.bin bs=1M count=10 status=none
ls -lh written-lab.bin
du -h written-lab.bin

Both files have a logical length of ten mebibytes. That is the length shown by ls. The first is sparse: parts of its length are unwritten holes that may require little actual storage. The dd command writes zeros across the second file and asks the filesystem to allocate data blocks for them.

Put the ls and du readings side by side. One describes length; the other describes allocated usage. Keep the numbers your own filesystem reports, because compression, copy-on-write and allocation rules can affect them. The comparison can reveal why a large-looking file would not necessarily return that much room when removed.

Those holes are still readable. Ask for bytes inside one and the filesystem can return zeros without having stored a separate block for every unwritten part. The file can therefore offer a long span of readable positions while occupying much less space.

A copy may change that arrangement. A copying or archiving tool might preserve the holes, or write their zeros out and allocate more storage. The apparent length can stay the same while the amount of occupied room changes. That is why the two readings are worth keeping separate.

The logger still has the file open

Now consider an illustrative logger with a two-gigabyte log open. Someone removes the filename, but the logger continues writing through the handle it already obtained, called a file descriptor. Our small lab does not create that large file; the example explains another reason a cleanup may leave the filesystem full.

The removed name is no longer there for du to find. The file’s blocks are still occupied, so df continues to count them. If lsof is installed, sudo lsof +L1 can help identify open files with no remaining links.

Finding the process tells you where to investigate, not which service to restart immediately. Identify the service and its supported log-rotation or reopen procedure. The aim is to release the old handle while preserving the service’s work.

The filename helped the logger open the file; the open handle let it keep using that file after the name vanished. On Linux, a file with no remaining links stays around while a process still has it open. Closing the last such descriptor allows the space to be reclaimed. Removing the name and ending the use are two events, and they need not happen together.

A cleanup has to help the filesystem that is full

Suppose /var has run out of bytes, still has plenty of inodes, and sits on a different mount from your home folder. Deleting the lab files at home will not make room in /var. If the shortage were inodes instead, a directory full of tiny cache files could be more relevant than the largest file on the machine.

For this exercise, remove only sparse-lab.bin and written-lab.bin, then repeat du. Record the path you examined, the filesystem containing it and the reading that explained the shortage. If those details do not fit together, investigate before deciding what else to remove.

The repeated “disk full” message now has several possible explanations you can distinguish. You know where to look for the space, why a file’s length is not its allocation, and how a logger can keep using a file whose name has gone. The next cleanup can begin with the write that needs room, rather than whichever file happens to look largest.

Storage counts and removed filenames

The GNU Coreutils manual documents df, du and truncate. The Linux unlink manual explains the lifetime of a file that is still open after its name has been removed.