looking for more uild hosts / Re: jenkins build slaves: disks full

historical

On Wed, Dec 05, 2018 at 04:22:07PM +0100, Neels Hofmeyr wrote:
> We recently hit a bunch of jenkins failures, due to a full disk.

thanks for investigating.

> Just now I removed 172G worth of docker images from build2-deb9build-ansible;
> I thought we had the docker cleanup automated by now?

I was also under that impression.

> The tmp/ has many folders like
> 196M	tmp.u3y02wgBNI
> which are all from March to May this year. I will delete them now.

I think that's the ttcn3 test runs, which (from our shell script?) generate
such a tmp directory for pcap and log files?

> Even after the docker cleanup commands I used from the osmocom.org servers wiki page:
> docker rm $(docker ps -a -q)
> docker rmi $(docker images -q -f dangling=true)
> There are still 321 docker images around, most of which are months old.
> Not sure why above cleanups don't catch those.

because something still references them.

> I'm just going to indiscriminately blow all of them away now.

would have been interesting to investigate where those references come from.  Maybe
something is starting containers that keep persistent state ('docker run' without '--rm'),
which will introduce reference counts to the underlying images?

> Maybe a good cleanup strategy would be to every week or so automatically wipe
> out the entire build slave lxc and re-create it from scratch?

an option, but not sure what kind of fall-out that might create due to build slave inavailability
during re-creation time, ...?

> On host-2 though there are a lot of services running.

sure, it's our main *.osmocom.org server.  NEver intended as a build host, just created
build slaves there as long as we still have capacity.

> ----- admin-2
> 
> It seems I can not login there, or at least I don't know the IP address...

Not even an osmocom.org machine.  Again just using some spare capacity there.

General note:

"large" root severs are unfortunately rather expensive to rent (even at Hetzner),
so I've been hesitant to spend even more than the several hundred EUR that we spend
for hosting per month already.

If somebody has spare capacity (mainly CPU + RAM) and would want to make that available
to Osmocom, it would be much appreciated.

Another strategy might be to simply buy machines and run them e.g. from
the sysmocom office.  The bandwidth requirements are low, so running
behind a cable modem is feasible.

Regards,
	Harald

-- 
- Harald Welte <laforge at gnumonks.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)