[PATCH] osmo-ttcn3-hacks[master]: support ccache to speed up ttcn3 test compilation

This is merely a historical archive of years 2008-2021, before the migration to mailman3.

A maintained and still updated list archive can be found at https://lists.osmocom.org/hyperkitty/list/gerrit-log@lists.osmocom.org/.

Stefan Sperling gerrit-no-reply at lists.osmocom.org
Fri Apr 6 13:07:13 UTC 2018


Hello Neels Hofmeyr, Harald Welte, Jenkins Builder,

I'd like you to reexamine a change.  Please visit

    https://gerrit.osmocom.org/7601

to look at the new patch set (#6).

support ccache to speed up ttcn3 test compilation

By default, if ccache is found in PATH when regen-makefile.sh runs,
use cached compiled C++ object files during the build (or populate
the cache in case an object is not cached yet). Objects are cached
in ccache's default cache directory (~/.ccache) unless overriden by
the CCACHE_DIR environment variable.

The USE_CCACHE environment variable overrides auto-detection
in regen-makefile.sh and disables (USE_CCACHE=0) or enables
(USE_CCACHE=1) use of ccache.

Rebuilding the test suite from scratch with a populated cache
is an order of magnitude faster than doing an initial build.
ccache only speeds up C++ (.cc) to object file (.o) compilation.
Linking object (.o) files into shared object (.so) files is still
performed by the real compiler during every build.

There is a small downside: We need to tweak the generated C++ files
to prevent cache misses due to timestamps inserted into the code
by ttcn3_compiler. This is done during 'make compile' if ccache
is used. This mechanism could break in case the output of
ttcn3_makefilegen changes. However, the only consequence would
be a slow build due to cache misses. In the long term, an upstream
solution built into eclipse titan (e.g. an option to disable time
stamps in generated files) would be diserable.

Another thing to note is that builds using objects from ccache
will contain timestamp strings from the build which populated
the cache. ccache provides a way to repopulate the cache if
needed (set CCACHE_RECACHE=1 in the environment).

These tradeoffs seem acceptable to me since in return we can
achieve much faster turnaround, both during local test
development and during test builds on jenkins.

Change-Id: Ibb538f602206535c06980f88191c1dabe3c4cd82
---
M regen-makefile.sh
A strip-datetime-comments.sed
2 files changed, 63 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.osmocom.org:29418/osmo-ttcn3-hacks refs/changes/01/7601/6

diff --git a/regen-makefile.sh b/regen-makefile.sh
index 57eebff..cd7a589 100755
--- a/regen-makefile.sh
+++ b/regen-makefile.sh
@@ -12,6 +12,16 @@
 
 test -x "$(which ttcn3_makefilegen 2>/dev/null)" || { echo "ERROR: ttcn3_makefilegen not in PATH"; exit 1; }
 
+# Enable ccache if it can be found in path.
+# This speeds up repeated builds of the TTCN3 tests by an order of magnitude
+# since most of the generated C++ source files don't change very often.
+# Roughly, for an initial build which takes N minutes, a complete rebuild
+# after 'make clean' will only take N seconds with ccache.
+# Note that ccache cannot speed up compilation of .o files to .so files.
+if [ -z "$USE_CCACHE" ] && which ccache 2>/dev/null; then
+	USE_CCACHE=1
+fi
+
 ttcn3_makefilegen -p -l -f $*
 sed -i -e 's/# TTCN3_DIR = /TTCN3_DIR = \/usr/' Makefile
 sed -i -e 's/LDFLAGS = /LDFLAGS = -L \/usr\/lib\/titan /' Makefile
@@ -32,3 +42,11 @@
         sed -i -e 's/TTCN3_DIR = $/TTCN3_DIR = \/usr/' Makefile
 fi
 sed -i -e 's/\/bin\/compiler/\/bin\/ttcn3_compiler/' Makefile
+
+if [ "x$USE_CCACHE" = "x1" ]; then
+	# enable ccache
+	sed -i -e 's/^CXX = g++ $/CXX = env CCACHE_SLOPPINESS=time_macros ccache g++/' Makefile
+	# The Makefile must tweak generated C++ files during the 'make compile' step.
+	# See the strip-datetime-comments sed script for details.
+	sed -i -f ../strip-datetime-comments.sed Makefile
+fi
diff --git a/strip-datetime-comments.sed b/strip-datetime-comments.sed
new file mode 100644
index 0000000..8089b8e
--- /dev/null
+++ b/strip-datetime-comments.sed
@@ -0,0 +1,45 @@
+#!/usr/bin/sed
+# (C) 2018 by sysmocom s.f.m.c. GmbH <info at sysmocom.de>
+# Author: Stefan Sperling <ssperling at sysmocom.de>
+# All rights reserved.
+# Released under the terms of GNU General Public License, Version 2 or
+# (at your option) any later version.
+
+# This sed script edits a Makefile generated by ttcn3_makefilegen.
+# We insert a sed command into the 'compile:' target in order to
+# remove timestamp comments from generated C++ files.
+
+# The ttcn3_compiler will generate C++ comments with a timestamp, such as:
+#   // for Stefan Sperling (stsp at fintan) on Sat Mar 31 14:34:30 2018
+# We must remove such comments since they cause cache misses with ccache.
+#
+# A related problem is the use of __DATE__ and __TIME__ in generated C++ code,
+# which can be worked around by setting "CCACHE_SLOPPINESS=time_macros". This
+# workaround implies that we must get our cache hits from ccache's "direct"
+# cache, since ccache's "preprocessor" caching method won't work.
+#
+# FIXME: This should really be addressed in ttcn3_compiler itself!
+# It should provide an option to suppress use of timestamps in generated code.
+
+# The generated Makefile's compile target looks like this:
+#
+# compile: $(TTCN3_MODULES)  $(PREPROCESSED_TTCN3_MODULES) $(ASN1_MODULES)
+# 	$(TTCN3_DIR)/bin/ttcn3_compiler $(COMPILER_FLAGS) \
+# 	$(TTCN3_MODULES) $(PREPROCESSED_TTCN3_MODULES) $(ASN1_MODULES) - $?
+# 	touch $@
+#
+# Before the final 'touch $@' step, we now insert a `sed` command that strips
+# offending C++ comments from the generated code.
+
+# match the line in the generated Makefile which starts with 'compile:'
+/^compile:/ {
+	# go to the next line
+	n;
+	# go to the next line
+	n;
+	# go to the next line
+	n;
+	# on this line, try to match 'touch $@', and if there is a match preprend a
+	# sed command which deletes the offending comments from generated C++ files:
+	s!touch $@!sed -i -e '/\\/\\/ for .* on .*/d' $(GENERATED_HEADERS) $(GENERATED_SOURCES) \&\& &!
+}

-- 
To view, visit https://gerrit.osmocom.org/7601
To unsubscribe, visit https://gerrit.osmocom.org/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb538f602206535c06980f88191c1dabe3c4cd82
Gerrit-PatchSet: 6
Gerrit-Project: osmo-ttcn3-hacks
Gerrit-Branch: master
Gerrit-Owner: Stefan Sperling <ssperling at sysmocom.de>
Gerrit-Reviewer: Harald Welte <laforge at gnumonks.org>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: Neels Hofmeyr <nhofmeyr at sysmocom.de>
Gerrit-Reviewer: Stefan Sperling <ssperling at sysmocom.de>



More information about the gerrit-log mailing list