osmith submitted this change.

View Change

testenv: optionally compress big ttcn3 logs

Use pigz (parallel implementation of gzip) to compress logs bigger than
1 MiB if TESTENV_COMPRESS_BIG_LOGS=1 is set, which will be the case in
jenkins. This saves a lot of space in the build artifacts.

I decided to not compress small log files because it can be useful to
quickly open them in the browser.

Related: OS#7105
Change-Id: I0766f6236f023ea8e3329c1364c9ee048e00c6d0
---
M _testenv/README.md
M _testenv/data/podman/Dockerfile
A _testenv/data/scripts/log_compress.sh
M _testenv/testenv/testsuite.py
4 files changed, 31 insertions(+), 0 deletions(-)

diff --git a/_testenv/README.md b/_testenv/README.md
index f10d349..4f53724 100644
--- a/_testenv/README.md
+++ b/_testenv/README.md
@@ -226,6 +226,11 @@
   available but doesn't work in podman. QEMU runs a bit slower when this is
   set.
 
+* `TESTENV_COMPRESS_BIG_LOGS`:
+  When set to 1, compress `.merged` ttcn3 log files that are bigger than 1 MiB
+  with `gz`. This is used in jenkins to reduce job artifact sizes. Requires
+  `pigz` to be installed when running without `--podman`.
+
 ### Environment variables set by testenv
 
 Testenv sets the following variables while running shell commands from
diff --git a/_testenv/data/podman/Dockerfile b/_testenv/data/podman/Dockerfile
index 6a1a42a..f808433 100644
--- a/_testenv/data/podman/Dockerfile
+++ b/_testenv/data/podman/Dockerfile
@@ -74,6 +74,7 @@
 		netcat-openbsd \
 		pax-utils \
 		pcscd \
+		pigz \
 		pkg-config \
 		procps \
 		psmisc \
diff --git a/_testenv/data/scripts/log_compress.sh b/_testenv/data/scripts/log_compress.sh
new file mode 100755
index 0000000..3e142ad
--- /dev/null
+++ b/_testenv/data/scripts/log_compress.sh
@@ -0,0 +1,13 @@
+#!/bin/sh -e
+# Compress big log files (OS#7105)
+min=1000000  # ~ 1 MiB
+
+for i in *.merged; do
+	size="$(stat -c"%s" "$i")"
+
+	if [ $size -lt $min ]; then
+		continue
+	fi
+
+	pigz -9 "$i"
+done
diff --git a/_testenv/testenv/testsuite.py b/_testenv/testenv/testsuite.py
index 52886cf..fc1716f 100644
--- a/_testenv/testenv/testsuite.py
+++ b/_testenv/testenv/testsuite.py
@@ -97,6 +97,17 @@
     testenv.cmd.run(cmd, cwd=cwd, env=builddir_env)
 
 
+def compress_big_log_files(cfg):
+    cwd = os.path.join(testenv.testdir.testdir, "testsuite")
+
+    if os.getenv("TESTENV_COMPRESS_BIG_LOGS") != "1":
+        return
+
+    logging.info("Compressing big log files")
+    cmd = os.path.join(testenv.data_dir, "scripts/log_compress.sh")
+    testenv.cmd.run(cmd, cwd=cwd, env=builddir_env)
+
+
 def get_junit_logs(topdir):
     pattern = os.path.join(topdir, "**", "junit-*.log")
     return glob.glob(pattern, recursive=True)
@@ -192,6 +203,7 @@
 
     merge_log_files(cfg)
     format_log_files(cfg)
+    compress_big_log_files(cfg)
 
     if testenv.args.bisect and not check_testsuite_successful():
         raise testenv.NoTraceException("Testsuite failed!")

To view, visit change 43896. To unsubscribe, or for help writing mail filters, visit settings.