docker build slaves / docker in lxc

Hi all, we just ran out of storage on one of our build slaves. /var/lib/docker inside the debian9 lxc had grown to 240 GB while actually it should be much smaller. "docker system prune" helped to freee 35GB of build cache, but still the remaining 217 GB was way too much compared to what it should use as per "docker system df -v" - and also much more than other build slaves were using (~ 13GB) It turned out that the files were in /var/lib/docker/vfs, and as I didn't see that yet on other systems, I investigated: * if the physical host hosting the lxc container doesn't have 'overlay.ko' loaded, overlay/overlay2 storage drivers are not available to docker * docker "silently" falls back to using "vfs" which is basically not copy-on-write but copy-everything-all-the-time, and which consumes massive amounts of storage I now made sure build2.osmocom.org loads "overlay.ko" by default and I changed the storage driver on the debian9 lxc. This however means you loos all containers/images, and everything has to be rebuilt from scratch. How to test: $ docker info | grep Storage should render Storage Driver: overlay2 If it shows vfs, something is wrong. I checked our other build slaves and they're set up correctly. Regards, Harald -- - Harald Welte <hwelte(a)sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Director: Harald Welte