Out Of Inodes

The Symptoms

You might be getting errors that make it sound like there is no space left on one of your servers disks. You might see messages like “ENOSPC” or “No space left on device” or any other generic error that essentially says you’re low on disk space, however when you run a command such as df -ah to check your disk usage you find that you have plenty of space.

My Specific Issue

For me this happened when trying to deploy a node application to a staging server on AWS. When it got to the point in the script where npm would pull down and install the package dependencies I started getting errors that made it sound like I had run out of disk space but when I checked and found that this wasn’t the case I was initially very confused as to what the issue was.

The Cause

Use df -ih to check your inode usage. Look for the percentage next to the disk you’re having the issue with, is it 99%? 100%? then that’s your issue. If you’re unsure what line of output you should be looking at it will probably be mounted on root (/) and be called something like /dev/xvda1 or /dev/sdb1 but this is dependent on your specific setup.

What Are Inodes?

In Linux, when you create a file or directory on the disk (as directories are secretly just a special type of file) you use a data block to store the data and an inode to store meta information about the file (such as owner, group, permissions, last access time, etc). The number of inodes is set at the point you partition a disk and can’t be changed post creation. If you run out of inodes then you can’t create any more files until you free up some inodes (just like when you run out of disk space) If you’re interested in a deeper dive on inodes read the articles below.

• Detailed Understanding Of Linux inodes With Example
• Linux Programmer’s Manual – INODE

Note: Btrfs and XFS use dynamic inodes so shouldn’t experience this issue, in addition ZFS does not use inodes (source).

The Solution

Tracking Down The Issue

Find inode usage in the current directory (start at root and work your way down until you find the source of the usage).

find . -maxdepth 1 -type d | grep -v '^\.$' | xargs -n 1 -i{} find {} -xdev -type f | cut -d "/" -f 2 | uniq -c | sort -n

For a full list use the command below (source). Note that this will take some time to run but the results are much more useful in my opinion.

sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

Note: I used these commands on Ubuntu Server 14.04 and Ubuntu Desktop 18.04.

What Fixed It For Me

In my case the issue was that there were a large number of old Linux headers that hadn’t been removed when new ones had been installed. The fix was therefore as simple as running the autoremove command. At the time the server was running Ubuntu Server 14.04 and was running on a 20GB disk.“`sudo apt-get autoremove“`

Edit: I’ve seen this issue more recently with some php session files not getting cleared out, ideally you would fix this by making them automatically clear but thought I would share this handy fix as a short term solution since it could come in handy in other situations. The below will allow you to delete files older than 30 days in a directory:

find /directory/goes/here -type f -mtime +30 -delete

Some Other Posts Dealing With Inodes

These are posts from other sites that I have found useful when resolving inode related issues. If this post hasn’t helped you then you might have some luck reading one of these.

• No Space Left On Device – Running Out Of Inodes
• What To Do When A Linux System Is Running Out Of Available Inodes
• Increasing The Amount Of Inotify Watchers
• Gulp Watch Error (ENOSPC)