[PATCH] osmo-ttcn3-hacks[master]: support ccache to speed up ttcn3 test compilation

This is merely a historical archive of years 2008-2021, before the migration to mailman3.

A maintained and still updated list archive can be found at https://lists.osmocom.org/hyperkitty/list/gerrit-log@lists.osmocom.org/.

Stefan Sperling gerrit-no-reply at lists.osmocom.org
Sun Apr 1 10:43:22 UTC 2018


Hello Neels Hofmeyr, Jenkins Builder,

I'd like you to reexamine a change.  Please visit

    https://gerrit.osmocom.org/7601

to look at the new patch set (#3).

support ccache to speed up ttcn3 test compilation

If ccache exists in PATH, use it to cache compiled C++ object files.

Rebuilding the test suite from scratch with a populated cache
is an order of magnitude faster than doing an initial build.
In my case, compile time for BSC_Tests goes down from 3 minutes
and 12 seconds to just 3 seconds, after 'make clean'.

There is a small downside: We need to tweak the generated C++ files
to prevent cache misses due to timestamps inserted into the code
by ttcn3_compiler. This is done by regen-makefile.sh if ccache
is used. This mechanism could break in case the output of
ttcn3_makefilegen changes. However, the only consequence would
be a slow build due to cache misses.

Another thing to note is that builds using objects from ccache
will contain timestamp strings from the build which populated
the cache. ccache provides a way to repopulate the cache if
needed though (set CCACHE_RECACHE=1 in the environment).

These tradeoffs seem acceptable to me since in return we can
achieve much faster turnaround, both during local test
development and during test runs on jenkins.

Change-Id: Ibb538f602206535c06980f88191c1dabe3c4cd82
---
M regen-makefile.sh
A strip-datetime-comments.sed
2 files changed, 63 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/01/7601/3

diff --git a/regen-makefile.sh b/regen-makefile.sh
index 57eebff..2d09e63 100755
--- a/regen-makefile.sh
+++ b/regen-makefile.sh
@@ -12,7 +12,22 @@
 
 test -x "$(which ttcn3_makefilegen 2>/dev/null)" || { echo "ERROR: ttcn3_makefilegen not in PATH"; exit 1; }
 
-ttcn3_makefilegen -p -l -f $*
+# Enable ccache if it can be found in path.
+# This speeds up repeated builds of the TTCN3 tests by an order of magnitude
+# since most of the generated C++ source files don't change very often.
+# Roughly, for an initial build which takes N minutes, a complete rebuild
+# after 'make clean' will only take N seconds with ccache.
+# Note that ccache cannot cache compilation of .o files to .so files, so do
+# not pass the -l option to ttcn3_makefilegen if ccache is used. The time
+# savings promised by the -l option are dwarfed by use of ccache anyway.
+if which ccache 2>/dev/null; then
+	USE_CCACHE=1
+	MAKEFILEGEN_FLAGS="-p -f"
+else
+	MAKEFILEGEN_FLAGS="-p -l -f"
+fi
+
+ttcn3_makefilegen $MAKEFILEGEN_FLAGS $*
 sed -i -e 's/# TTCN3_DIR = /TTCN3_DIR = \/usr/' Makefile
 sed -i -e 's/LDFLAGS = /LDFLAGS = -L \/usr\/lib\/titan /' Makefile
 #sed -i -e 's/TTCN3_LIB = ttcn3-parallel/TTCN3_LIB = ttcn3/' Makefile
@@ -32,3 +47,11 @@
         sed -i -e 's/TTCN3_DIR = $/TTCN3_DIR = \/usr/' Makefile
 fi
 sed -i -e 's/\/bin\/compiler/\/bin\/ttcn3_compiler/' Makefile
+
+if [ -n "$USE_CCACHE" ]; then
+	# enable ccache
+	sed -i -e 's/^CXX = g++ $/CXX = env CCACHE_SLOPPINESS=time_macros ccache g++/' Makefile
+	# The Makefile must tweak generated C++ files during the 'make compile' step.
+	# See the strip-datetime-comments sed script for details.
+	sed -i -f ../strip-datetime-comments.sed Makefile
+fi
diff --git a/strip-datetime-comments.sed b/strip-datetime-comments.sed
new file mode 100644
index 0000000..217dedc
--- /dev/null
+++ b/strip-datetime-comments.sed
@@ -0,0 +1,39 @@
+#!/usr/bin/sed
+# This sed script edits a Makefile generated by ttcn3_makefilegen.
+# We insert a sed command into the 'compile:' target in order to
+# remove timestamp comments from generated C++ files.
+
+# The ttcn3_compiler will generate C++ comments with a timestamp, such as:
+#   // for Stefan Sperling (stsp at fintan) on Sat Mar 31 14:34:30 2018
+# We must remove such comments since they cause cache misses with ccache.
+#
+# A related problem is the use of __DATE__ and __TIME__ in generated C++ code,
+# which can be worked around by setting "CCACHE_SLOPPINESS=time_macros". This
+# workaround implies that we must get our cache hits from ccache's "direct"
+# cache, since ccache's "preprocessor" caching method won't work.
+#
+# FIXME: This should really be addressed in ttcn3_compiler itself!
+# It should provide an option to suppress use of timestamps in generated code.
+
+# The generated Makefile's compile target looks like this:
+#
+# compile: $(TTCN3_MODULES)  $(PREPROCESSED_TTCN3_MODULES) $(ASN1_MODULES)
+# 	$(TTCN3_DIR)/bin/ttcn3_compiler $(COMPILER_FLAGS) \
+# 	$(TTCN3_MODULES) $(PREPROCESSED_TTCN3_MODULES) $(ASN1_MODULES) - $?
+# 	touch $@
+#
+# Before the final 'touch $@' step, we now insert a `sed` command that strips
+# offending C++ comments from the generated code.
+
+# match the line in the generated Makefile which starts with 'compile:'
+/^compile:/ {
+	# go to the next line
+	n;
+	# go to the next line
+	n;
+	# go to the next line
+	n;
+	# on this line, try to match 'touch $@', and if there is a match preprend a
+	# sed command which deletes the offending comments from generated C++ files:
+	s!touch $@!sed -i -e '/\\/\\/ for .* on .*/d' $(GENERATED_HEADERS) $(GENERATED_SOURCES) \&\& &!
+}

-- 
To view, visit https://gerrit.osmocom.org/7601
To unsubscribe, visit https://gerrit.osmocom.org/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb538f602206535c06980f88191c1dabe3c4cd82
Gerrit-PatchSet: 3
Gerrit-Project: osmo-ttcn3-hacks
Gerrit-Branch: master
Gerrit-Owner: Stefan Sperling <ssperling at sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: Neels Hofmeyr <nhofmeyr at sysmocom.de>
Gerrit-Reviewer: Stefan Sperling <ssperling at sysmocom.de>



More information about the gerrit-log mailing list