Disk Full Investigation in Linux

A Linux server running out of disk space is one of the most common operational issues faced by system administrators and DevOps engineers. Whether it's an appli...

A Linux server running out of disk space is one of the most common operational issues faced by system administrators and DevOps engineers. Whether it's an application crash, failed deployments, database errors, or a server refusing to boot, a full disk can quickly turn into a production incident.
Why Disk Space Matters
When a filesystem reaches 100% utilization, several problems can occur:

  • Applications fail to write data.
  • Databases may stop accepting transactions.
  • System logs stop updating.
  • Package installations fail.
  • Temporary files cannot be created.
  • Performance may degrade significantly.

Instead of deleting files randomly, it's important to identify what is consuming the space.


Step 1: Check Filesystem Usage
The first ever command will show complete disk usage is:

df -h

img_1784440743_1e1155cf_image.webp
Understanding the output

  • Size – Total filesystem size
  • Used – Space currently used
  • Avail – Remaining free space
  • Use% – Percentage utilized
  • Mounted on – Mount point

If / shows 100%, further investigation is required.


Step 2: Identify Large Directories
Move to the affected filesystem and execute:

du -sh /*

img_1784441487_6e41d73f_image.webp
This will show all file system directory space.
For a more detailed view:

du -xh / --max-depth=1 | sort -hr

img_1784441779_dbab707a_image.webp
--max-depth option limits the depth of directory recursion when using the du (disk usage) command. It stops the terminal from flooding your screen with thousands of nested subdirectories, allowing you to quickly isolate which top-level folders are hogging your storage space.


Step 3: Drill Down into Large Directories
Suppose /var is consuming most of the storage.

du -xh /var --max-depth=1 | sort -hr

img_1784442908_fdf4cc32_image.webp
Continue drilling down until you locate the exact directory.


Step 4: Find the Largest Files
Locate files larger than 500 MB:

find / -type f -size +500M 2>/dev/null

Or list the largest files:

find / -type f -exec ls -lh {} \; 2>/dev/null | sort -k5 -hr | head -20

Typical offenders include:

  • Large log files
  • Database dumps
  • Backup archives
  • Core dump files
  • Temporary exports

Step 5: Check Log Files
Logs are one of the biggest reasons disks become full.Check log directory size:

du -sh /var/log

Find the largest logs:

find /var/log -type f -exec ls -lh {} \; | sort -k5 -hr | head

Check systemd journal size:

journalctl --disk-usage

Reduce journal size:

journalctl --vacuum-size=500M

Step 6: Investigate Deleted but Open Files
Sometimes a file has been deleted, yet its space is not released because a running process still holds it open.
Check:

lsof | grep deleted

Example:

java    2314 appuser  10w REG 8,1 5.2G deleted

Restart the application or service to release the storage.


Step 7: Inspect Docker (If Installed)
Docker frequently consumes significant storage.
View usage:

docker system df

Clean unused resources:

docker system prune

Remove unused images:

docker image prune -a

Step 8: Check Package Cache
Ubuntu/Debian:

sudo apt autoremove -y
sudo apt autoclean

RHEL/CentOS:

sudo yum clean all
sudo dnf clean all

Step 9: Analyze Disk Usage Interactively
A favorite tool among Linux administrators is ncdu.
Ubuntu:

sudo apt install ncdu

RHEL:

sudo yum install ncdu

Run:

ncdu /

img_1784445412_6d36ee15_image.webp

It provides an interactive interface for browsing large directories and identifying storage-heavy files.


Common Causes of Full Disks

CauseSymptomsSolution
Log files/var/log grows rapidlyRotate or truncate logs
Docker imagesLarge /var/lib/dockerPrune unused images and containers
BackupsOld .tar, .gz, .zip filesArchive or remove obsolete backups
DatabasesGrowing data filesArchive data, optimize storage
Core dumpsLarge core.* filesRemove old dumps, investigate crashes
Temporary files/tmp or /var/tmp fills upClean unused temporary files

Share This Post

Latest Comments (0)

No comments yet. Be the first to comment!