osmith has submitted this change. ( https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/43896?usp=email ) Change subject: testenv: optionally compress big ttcn3 logs ...................................................................... testenv: optionally compress big ttcn3 logs Use pigz (parallel implementation of gzip) to compress logs bigger than 1 MiB if TESTENV_COMPRESS_BIG_LOGS=1 is set, which will be the case in jenkins. This saves a lot of space in the build artifacts. I decided to not compress small log files because it can be useful to quickly open them in the browser. Related: OS#7105 Change-Id: I0766f6236f023ea8e3329c1364c9ee048e00c6d0 --- M _testenv/README.md M _testenv/data/podman/Dockerfile A _testenv/data/scripts/log_compress.sh M _testenv/testenv/testsuite.py 4 files changed, 31 insertions(+), 0 deletions(-) Approvals: Jenkins Builder: Verified fixeria: Looks good to me, approved pespin: Looks good to me, but someone else must approve diff --git a/_testenv/README.md b/_testenv/README.md index f10d349..4f53724 100644 --- a/_testenv/README.md +++ b/_testenv/README.md @@ -226,6 +226,11 @@ available but doesn't work in podman. QEMU runs a bit slower when this is set. +* `TESTENV_COMPRESS_BIG_LOGS`: + When set to 1, compress `.merged` ttcn3 log files that are bigger than 1 MiB + with `gz`. This is used in jenkins to reduce job artifact sizes. Requires + `pigz` to be installed when running without `--podman`. + ### Environment variables set by testenv Testenv sets the following variables while running shell commands from diff --git a/_testenv/data/podman/Dockerfile b/_testenv/data/podman/Dockerfile index 6a1a42a..f808433 100644 --- a/_testenv/data/podman/Dockerfile +++ b/_testenv/data/podman/Dockerfile @@ -74,6 +74,7 @@ netcat-openbsd \ pax-utils \ pcscd \ + pigz \ pkg-config \ procps \ psmisc \ diff --git a/_testenv/data/scripts/log_compress.sh b/_testenv/data/scripts/log_compress.sh new file mode 100755 index 0000000..3e142ad --- /dev/null +++ b/_testenv/data/scripts/log_compress.sh @@ -0,0 +1,13 @@ +#!/bin/sh -e +# Compress big log files (OS#7105) +min=1000000 # ~ 1 MiB + +for i in *.merged; do + size="$(stat -c"%s" "$i")" + + if [ $size -lt $min ]; then + continue + fi + + pigz -9 "$i" +done diff --git a/_testenv/testenv/testsuite.py b/_testenv/testenv/testsuite.py index 52886cf..fc1716f 100644 --- a/_testenv/testenv/testsuite.py +++ b/_testenv/testenv/testsuite.py @@ -97,6 +97,17 @@ testenv.cmd.run(cmd, cwd=cwd, env=builddir_env) +def compress_big_log_files(cfg): + cwd = os.path.join(testenv.testdir.testdir, "testsuite") + + if os.getenv("TESTENV_COMPRESS_BIG_LOGS") != "1": + return + + logging.info("Compressing big log files") + cmd = os.path.join(testenv.data_dir, "scripts/log_compress.sh") + testenv.cmd.run(cmd, cwd=cwd, env=builddir_env) + + def get_junit_logs(topdir): pattern = os.path.join(topdir, "**", "junit-*.log") return glob.glob(pattern, recursive=True) @@ -192,6 +203,7 @@ merge_log_files(cfg) format_log_files(cfg) + compress_big_log_files(cfg) if testenv.args.bisect and not check_testsuite_successful(): raise testenv.NoTraceException("Testsuite failed!") -- To view, visit https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/43896?usp=email To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: osmo-ttcn3-hacks Gerrit-Branch: master Gerrit-Change-Id: I0766f6236f023ea8e3329c1364c9ee048e00c6d0 Gerrit-Change-Number: 43896 Gerrit-PatchSet: 2 Gerrit-Owner: osmith <osmith@sysmocom.de> Gerrit-Reviewer: Jenkins Builder Gerrit-Reviewer: fixeria <vyanitskiy@sysmocom.de> Gerrit-Reviewer: osmith <osmith@sysmocom.de> Gerrit-Reviewer: pespin <pespin@sysmocom.de>