No space left on device

What happened if you had a big partition with a lot of free space but you are not able to create a file or copy anything there? The linux said “no space left on device”. The trivial answer was you’re out of inodes.

I didn’t think I will ever hear about this problem except the books. But yesterday by boss came to me with a familiar problem. So like any conscientious engineer I must reproduce this on my virtual machine to be able to see the exact things happening in a situation like this.

So I made a partition with 114GB size.

I copy a lot of files here. The data files of MySQL (with innodb_file_per_table option) were set to be stored here. All the sites from the /var/www directory was copied and the the new www directory was mounted with bind option back to /var/www. So we have a lot of files here. All right. What else should we do?

Nothing, this was enough to reach the limit.

By default Inode count was based on the block count / inode size. The block count can be derived from the size of the partition and the block size. In our case partition size was 121791053824 byte ~ 114GB. Divided by the 4096 block size we get 29734144 block count which means 116224 inode. What can we do with it. After we have made the partition we have only the option

If we have any spare space on somewhere on the disks the easiest method is to copy all the data from this partition to another. Remake the partition with a modified parameter or with other filesystem (for example xfs).

Now we have 118936576 inodes on the new filesystem. Everybody’s happy. We can mount it back. Move all the data back. And let’s roll…