I looked at why the 420G of the Osmocombuild1 root fs are full.
1)
Slave "build1-debian9-lxc" is running as lxc called 'docker'.
/var/lib/lxc/docker/rootfs/var/lib/docker/vfs/dir contains 221G in 333 docker
snapshots. It seems to me that these are growing indefinitely, that nothing is
removing docker hashes that have been built. Does anyone have a discarding
strategy ready that we can deploy there? We need to add one to avoid the build
slave filling up again.
https://osmocom.org/issues/2539
For now I did
docker rmi $(docker images -q -f dangling=true)
and rm'd a few stopped containers that had started an openbsc test:
docker rm $(docker ps -aq)
which removed hundreds of hashes and freed ~110G
But half of the 221G are still there, not sure if we need those.
I also note that this lxc is not starting automatically after I rebooted
Osmocombuild1 (to get rid of some stuck processes).
I added
lxc.start.auto = 1
to /var/lib/lxc/docker/config
Now the server is back up and running, the lxc is started, but the jenkins
slave refuses to connect. It tries port 45 but nothing is listening
there. I think I've hit this before but can't remember how to solve it :/
https://osmocom.org/issues/2540
2)
Slave "Osmocombuild1" is running as user osmocom-build on the host's OS
itself.
The workspaces make up one large chunk of the disk fill: 94G.
Looking at OpenBSC@3, it builds up to 3.8G from the various build matrix
workspaces, where each has a compiled libosmocore at >110MB and osmo-iuh
at >130MB. Then the parrallel builds each make an own workspace, exploding the
total. The good news is that it should cap *somewhere* and not grow
indefinitely.
We probably should adjust the OpenBSC and various gerrit jobs to clean up the
workspace after they are done. We so far never needed the binaries.
https://osmocom.org/issues/2538
~N