osmith has posted comments on this change. ( https://gerrit.osmocom.org/c/osmo-ci/+/30476 )
Change subject: ansible/podman: add cronjob to clean old images
......................................................................
Patch Set 4:
(1 comment)
Patchset:
PS4:
> we've had our fair share of race conditions with cron jobs deleting images while jenkins jobs execut […]
fixing that for docker here: https://gerrit.osmocom.org/q/topic:fix-docker-clean-up
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/30476
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: Ib3c7575943b842b6b5a15967c59fa8d0a9b2d6f5
Gerrit-Change-Number: 30476
Gerrit-PatchSet: 4
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: laforge <laforge(a)osmocom.org>
Gerrit-Reviewer: msuraev <msuraev(a)sysmocom.de>
Gerrit-Reviewer: pespin <pespin(a)sysmocom.de>
Gerrit-Comment-Date: Fri, 09 Dec 2022 09:42:48 +0000
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Comment-In-Reply-To: laforge <laforge(a)osmocom.org>
Gerrit-MessageType: comment
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/30526 )
Change subject: scripts/docker-cleanup: remove fallback code
......................................................................
scripts/docker-cleanup: remove fallback code
Remove the fallback clean up code, as it also may lead to images getting
removed right before we need to use them. Besides that, it should be
dead code by now since docuum should be running on all our jenkins nodes
to clean up old images based on last use date.
Change-Id: I9ca0c2ba245bdd75d9fb8eaf341055e8c2ab1b55
---
M scripts/docker-cleanup.sh
1 file changed, 0 insertions(+), 9 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/26/30526/1
diff --git a/scripts/docker-cleanup.sh b/scripts/docker-cleanup.sh
index 4d8241b..fe134d2 100755
--- a/scripts/docker-cleanup.sh
+++ b/scripts/docker-cleanup.sh
@@ -1,15 +1,6 @@
#!/bin/sh -x
# https://osmocom.org/projects/osmocom-servers/wiki/Docker_cache_clean_up
-# simple image cleaning code in case docuum isn't running
-# delete all but the latest images
-if [ -z "$(docker ps -q -f name=docuum)" ]; then
- IMAGES=`docker image ls | grep \^osmocom-build | grep -v latest | awk -F ' ' '{print $1":"$2}'`
- for f in $IMAGES; do
- docker image rm $f
- done
-fi
-
# delete all containers where we forgot to use --rm with docker run,
# older than 24 hours
docker container prune --filter "until=24h" -f
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/30526
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: I9ca0c2ba245bdd75d9fb8eaf341055e8c2ab1b55
Gerrit-Change-Number: 30526
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-MessageType: newchange
osmith has uploaded this change for review. ( https://gerrit.osmocom.org/c/osmo-ci/+/30525 )
Change subject: scripts/docker-cleanup: fix timing problems
......................................................................
scripts/docker-cleanup: fix timing problems
Don't delete images while they are being used, to fix these errors we
see from time to time in the middle of "docker build" on jenkins:
unknown parent image ID sha256:1b072e35048cd8b680eddabdc641ac678edb1184d222d5e7b3fbe0b3c333129a
This happens because "docker build" creates so-called dangling images
for each step processed of a Dockerfile. The "docker system prune" call
deletes these dangling images (among other things).
Remove the "docker system prune" call. We already have the docuum daemon
to deal with unused images (dangling and not dangling), it removes them
based on last use date so that the used space is always below a
configured limit. As it deletes images that haven't been used the
longest when it reaches the limit, it will not result in the problem
explained above.
Besides images, "docker system prune" also removes unused containers
(instances of images created with 'docker run' without --rm) and
networks. Add "docker container prune" and "docker network prune"
commands to remove them from now on.
Also remove the redundant container removal logic (previous it was
redundant with "docker system prune", now redundant with "docker
container prune").
Related: https://docs.docker.com/config/pruning/
Change-Id: Ia1b466eea43dd135373949e8e3e6b005c169ea0c
---
M scripts/docker-cleanup.sh
1 file changed, 5 insertions(+), 7 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-ci refs/changes/25/30525/1
diff --git a/scripts/docker-cleanup.sh b/scripts/docker-cleanup.sh
index 49b976b..4d8241b 100755
--- a/scripts/docker-cleanup.sh
+++ b/scripts/docker-cleanup.sh
@@ -10,11 +10,9 @@
done
fi
-# delete all containers where we forgot to use --rm with docker run
-CONTAINERS="$(docker ps -q -a -f status=exited -f status=created)"
-if [ -n "$CONTAINERS" ]; then
- docker rm $CONTAINERS
-fi
+# delete all containers where we forgot to use --rm with docker run,
+# older than 24 hours
+docker container prune --filter "until=24h" -f
-# remove dangling images, containers, volumes, and networks (not tagged or associated with a container)
-docker system prune -f
+# remove unused networks older than 24 hours
+docker network prune --filter "until=24h" -f
--
To view, visit https://gerrit.osmocom.org/c/osmo-ci/+/30525
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings
Gerrit-Project: osmo-ci
Gerrit-Branch: master
Gerrit-Change-Id: Ia1b466eea43dd135373949e8e3e6b005c169ea0c
Gerrit-Change-Number: 30525
Gerrit-PatchSet: 1
Gerrit-Owner: osmith <osmith(a)sysmocom.de>
Gerrit-MessageType: newchange