Investigating why my 2TB SSD was full after 15 GB
Published on by 0np, 736 words
I tried to update my Immich instance recently, but it failed due to missing disk space:
0np@homelab:~/services/immich$ sudo docker compose pull
[+] Pulling 12/43
#...snip...
write /var/lib/docker/tmp/GetImageBlob424233791: no space left on device
This is odd, because I have a 2TB NVMe SSD in there and I am nowhere near close to the limit. I know I can free up space using docker system prune, but this is a fresh install and I'm storing all things on my ZFS disks outside my NVMe.
Maybe I messed up something during installation? First, let's check the current disk filesystem usage with df
0np@homelab:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora-root 15G 15G 449M 98% /
We can also utilize lsblk to see the size of the individual partitions:
nvme0n1 259:0 0 1.8T 0 disk
├─nvme0n1p1 259:1 0 600M 0 part /boot/efi
├─nvme0n1p2 259:2 0 1G 0 part /boot
└─nvme0n1p3 259:3 0 1.8T 0 part
└─luks-8009f1b8-6233-4951-aa44-0ea4447b0cd1 253:0 0 1.8T 0 crypt
└─fedora-root 253:1 0 15G 0 lvm /
There it is. It is only 15GB in size. That explains why pulling the updated container images failed. I don't know what I selected back when installing Fedora on my homelab server. It was more or less on autopilot, and I thought I selected to just use the whole disk for storage.
Ok, time to boot up gparted to fix it. Luckily, I can also change the partition settings on encrypted partitions. But unfortunately it seems all right:
Researching this issue brought me to [0] and [1]. It seems this is the default behaviour due to partition details (physical vs logical volume, resizing issues with one another). This gives me slight PTSD flashbacks from my time at IT school sixteen years ago. Somewhere in the distance I hear a small bell ring.
Anyway, for my homelab I want to keep things as simple as possible, so I simply extended the size of the logical volume /dev/mapper/fedora-root to 1200GB for now.
0np@homelab:~$ sudo lvextend -r -L +1200G /dev/mapper/fedora-root
Size of logical volume fedora/root changed from <1.48 TiB (387840 extents) to 1.67 TiB (439040 extents).
File system xfs found on fedora/root mounted at /.
Extending file system xfs to 1.67 TiB (1841467228160 bytes) on fedora/root...
xfs_growfs /dev/fedora/root
meta-data=/dev/mapper/fedora-root isize=512 agcount=4, agsize=983040 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=1
= reflink=1 bigtime=1 inobtcount=1 nrext64=1
data = bsize=4096 blocks=3932160, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=16384, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 3932160 to 449576960
xfs_growfs done
Extended file system xfs on fedora/root.
Logical volume fedora/root successfully resized.
I initiality executed this without the -r option. While this did increase the size of the logical volume, it did not resize it. This means the now 1.67TiB might not be consistend with the 1200GB mentioned in this blog post.
One restart later, and we are golden:
0np@homelab:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora-root 1.7T 48G 1.7T 3% /
Now I can update Immich successfully again:
0np@homelab:~/services/immich$ sudo docker compose pull
[+] Pulling 40/40
✔ immich-machine-learning Pulled 18.3s
#...snip...
✔ immich-database Pulled 0.5s
✔ immich-ts Pulled 1.2s
✔ immich-redis Pulled 0.5s
✔ immich-server Pulled 52.7s
#...snip...