Hi all,
we just ran out of storage on one of our build slaves. /var/lib/docker
inside the debian9 lxc had grown to 240 GB while actually it should be
much smaller. "docker system prune" helped to freee 35GB of build cache,
but still the remaining 217 GB was way too much compared to what it should
use as per "docker system df -v" - and also much more than other build
slaves were using (~ 13GB)
It turned out that the files were in /var/lib/docker/vfs, and as I didn't
see that yet on other systems, I investigated:
* if the physical host hosting the lxc container doesn't have 'overlay.ko'
loaded, overlay/overlay2 storage drivers are not available to docker
* docker "silently" falls back to using "vfs" which is basically not
copy-on-write
but copy-everything-all-the-time, and which consumes massive amounts of storage
I now made sure
build2.osmocom.org loads "overlay.ko" by default and I
changed the storage driver on the debian9 lxc. This however means you
loos all containers/images, and everything has to be rebuilt from
scratch.
How to test:
$ docker info | grep Storage
should render
Storage Driver: overlay2
If it shows vfs, something is wrong. I checked our other build slaves
and they're set up correctly.
Regards,
Harald
--
- Harald Welte <hwelte(a)sysmocom.de>
http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte