<p>Pau Espin Pedrol <strong>merged</strong> this change.</p><p><a href="https://gerrit.osmocom.org/13511">View Change</a></p><div style="white-space:pre-wrap">Approvals:
Jenkins Builder: Verified
Pau Espin Pedrol: Looks good to me, approved
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">process: Prevent NetNSProcess alive forever after SIGKILL<br><br>NetNSProcess are run in the following process tree:<br>osmo-gsm-tester -> sudo -> bash (osmo-gsm-tester_netns_exec.sh) -><br>tcpdump.<br><br>Lots of osmo-gsm-tester_netns_exec.sh scripts with tcpdump child process<br>were spotted in prod setup of osmo-gsm-tester. Apparently that happens<br>because sometimes tcpdump doesn't get killed in time with SIGTERM and<br>SIGINT, and as a result SIGKILL is sent by osmo-gsm-tester as usual<br>termination procedure. When SIGKILL is sent, the parent sudo process is<br>instantly killed without possibility to forward the signal to its<br>children, leaving the bash script and tcpdump alive.<br><br>In order to fix it, catch SIGKILL for this process class and send<br>instead SIGUSR1. Then, modify the script under sudo to handle SIGUSR1 as<br>if it was a SIGKILL towards its children to make sure child process in<br>the netns terminates.<br><br>Change-Id: I2bf389c47bbbd75f46af413e7ba897be5be386e1<br>---<br>M src/osmo_gsm_tester/process.py<br>M utils/osmo-gsm-tester_netns_exec.sh<br>2 files changed, 42 insertions(+), 1 deletion(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/src/osmo_gsm_tester/process.py b/src/osmo_gsm_tester/process.py</span><br><span>index 7ecb67e..441d4ea 100644</span><br><span>--- a/src/osmo_gsm_tester/process.py</span><br><span>+++ b/src/osmo_gsm_tester/process.py</span><br><span>@@ -363,6 +363,11 @@</span><br><span> # HACK: Since we run under sudo, only way to kill root-owned process is to kill as root...</span><br><span> # This function is overwritten from Process.</span><br><span> def send_signal(self, sig):</span><br><span style="color: hsl(120, 100%, 40%);">+ if sig == signal.SIGKILL:</span><br><span style="color: hsl(120, 100%, 40%);">+ # if we kill sudo, its children (bash running NETNS_EXEC_BIN +</span><br><span style="color: hsl(120, 100%, 40%);">+ # tcpdump under it) are kept alive. Let's instead tell the script to</span><br><span style="color: hsl(120, 100%, 40%);">+ # kill tcpdump:</span><br><span style="color: hsl(120, 100%, 40%);">+ sig = signal.SIGUSR1</span><br><span> kill_cmd = ('kill', '-%d' % int(sig), str(self.process_obj.pid))</span><br><span> run_local_netns_sync(self.run_dir, self.name()+"-kill"+str(sig), self.netns, kill_cmd)</span><br><span> </span><br><span>diff --git a/utils/osmo-gsm-tester_netns_exec.sh b/utils/osmo-gsm-tester_netns_exec.sh</span><br><span>index 336b746..182ebff 100755</span><br><span>--- a/utils/osmo-gsm-tester_netns_exec.sh</span><br><span>+++ b/utils/osmo-gsm-tester_netns_exec.sh</span><br><span>@@ -1,5 +1,41 @@</span><br><span> #!/bin/bash</span><br><span> netns="$1"</span><br><span> shift</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+child_ps=0</span><br><span style="color: hsl(120, 100%, 40%);">+forward_kill() {</span><br><span style="color: hsl(120, 100%, 40%);">+ sig="$1"</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Caught signal SIG$sig!"</span><br><span style="color: hsl(120, 100%, 40%);">+ if [ "$child_ps" != "0" ]; then</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Killing $child_ps with SIG$sig!"</span><br><span style="color: hsl(120, 100%, 40%);">+ kill -SIG${sig} $child_ps</span><br><span style="color: hsl(120, 100%, 40%);">+ else</span><br><span style="color: hsl(120, 100%, 40%);">+ exit 0</span><br><span style="color: hsl(120, 100%, 40%);">+ fi</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+forward_kill_int() {</span><br><span style="color: hsl(120, 100%, 40%);">+ forward_kill "INT"</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+forward_kill_term() {</span><br><span style="color: hsl(120, 100%, 40%);">+ forward_kill "TERM"</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+forward_kill_usr1() {</span><br><span style="color: hsl(120, 100%, 40%);">+ # Special signal received from osmo-gsm-tester to tell child to SIGKILL</span><br><span style="color: hsl(120, 100%, 40%);">+ echo "Converting SIGUSR1->SIGKILL"</span><br><span style="color: hsl(120, 100%, 40%);">+ forward_kill "KILL"</span><br><span style="color: hsl(120, 100%, 40%);">+}</span><br><span style="color: hsl(120, 100%, 40%);">+# Don't use 'set -e', otherwise traps are not triggered!</span><br><span style="color: hsl(120, 100%, 40%);">+trap forward_kill_int INT</span><br><span style="color: hsl(120, 100%, 40%);">+trap forward_kill_term TERM</span><br><span style="color: hsl(120, 100%, 40%);">+trap forward_kill_usr1 USR1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> #TODO: Later on I may want to call myself with specific ENV and calling sudo in order to run inside the netns but with dropped privileges</span><br><span style="color: hsl(0, 100%, 40%);">-ip netns exec $netns "$@"</span><br><span style="color: hsl(120, 100%, 40%);">+ip netns exec $netns "$@" &</span><br><span style="color: hsl(120, 100%, 40%);">+child_ps=$!</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+echo "$$: waiting for $child_ps"</span><br><span style="color: hsl(120, 100%, 40%);">+wait "$child_ps"</span><br><span style="color: hsl(120, 100%, 40%);">+child_exit_code="$?"</span><br><span style="color: hsl(120, 100%, 40%);">+echo "child exited with $child_exit_code"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+exit $child_exit_code</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/13511">change 13511</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/13511"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: merged </div>
<div style="display:none"> Gerrit-Change-Id: I2bf389c47bbbd75f46af413e7ba897be5be386e1 </div>
<div style="display:none"> Gerrit-Change-Number: 13511 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: Pau Espin Pedrol <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder (1000002) </div>
<div style="display:none"> Gerrit-Reviewer: Pau Espin Pedrol <pespin@sysmocom.de> </div>