Change in osmo-gsm-tester[master]: remote: Improve wrapper script handling proper exit of ssh-remote pro...

[pespin]

pespin has submitted this change. ( https://gerrit.osmocom.org/c/osmo-gsm-tester/+/17125 )

Change subject: remote: Improve wrapper script handling proper exit of ssh-remote processes
......................................................................

remote: Improve wrapper script handling proper exit of ssh-remote processes

Improvements include:
* Avoid race condition between receiving signal and process not yet
  started
* Make sure process is killed with -9 if process is still alive a while after
  we killed it (SIGINT, SIGTERM). This is useful for processes which
  sometimes hang during shutdown like srsue in some conditions.

Change-Id: I3c656b008a3c2b2bb453a59e51d338cb272fa50b
---
M src/osmo_gsm_tester/remote.py
1 file changed, 41 insertions(+), 12 deletions(-)

Approvals:
  Jenkins Builder: Verified
  pespin: Looks good to me, approved



diff --git a/src/osmo_gsm_tester/remote.py b/src/osmo_gsm_tester/remote.py
index 771f2b8..65c621e 100644
--- a/src/osmo_gsm_tester/remote.py
+++ b/src/osmo_gsm_tester/remote.py
@@ -59,20 +59,49 @@
         wrapper_script = self.run_dir.new_file(RemoteHost.WRAPPER_SCRIPT)
         with open(wrapper_script, 'w') as f:
             r = """#!/bin/bash
-            mypid=0
-            sign_handler() {
-                    sig=$1
-                    echo "received signal handler $sig, killing $mypid"
-                    kill $mypid
+            LOGFILE=/tmp/yes
+            kill_pid(){
+                mypid=$1
+                kill $mypid
+                if ! kill -0 $mypid; then
+                    return
+                fi
+                echo "sleeping some time waiting for child to die..." >>$LOGFILE
+                sleep 5
+                if ! kill -0 $mypid; then
+                    return
+                fi
+                echo "kill -9 the process and wait!" >>$LOGFILE
+                kill -9 $mypid
+                wait $mypid
             }
-            trap 'sign_handler SIGTERM' SIGTERM
-            trap 'sign_handler SIGINT' SIGINT
-            trap 'sign_handler SIGHUP' SIGHUP
+            prep_sighandler() {
+                unset term_child_pid
+                unset term_kill_needed
+                trap 'sign_handler SIGTERM' SIGTERM
+                trap 'sign_handler SIGINT' SIGINT
+                trap 'sign_handler SIGHUP' SIGHUP
+                echo "script started, traps set" >$LOGFILE
+            }
+            sign_handler() {
+                sig=$1
+                echo "$sig -> ${term_child_pid}" >>$LOGFILE
+                echo "received signal handler $sig, killing ${term_child_pid}" >>$LOGFILE
+                kill_pid ${term_child_pid}
+            }
+            wait_sighandler()
+            {
+                term_child_pid=$!
+                if [ "${term_kill_needed}" ]; then
+                    kill_pid "${term_child_pid}"
+                fi
+                echo "waiting for ${term_child_pid}" >>$LOGFILE
+                wait ${term_child_pid}
+                echo "process ${term_child_pid} finished" >>$LOGFILE
+            }
+            prep_sighandler
             $@ &
-            mypid=$!
-            echo "waiting for $mypid"
-            wait $mypid
-            echo "process $mypid finished"
+            wait_sighandler
             """
             f.write(r)
         st = os.stat(wrapper_script)

-- 
To view, visit https://gerrit.osmocom.org/c/osmo-gsm-tester/+/17125
To unsubscribe, or for help writing mail filters, visit https://gerrit.osmocom.org/settings

Gerrit-Project: osmo-gsm-tester
Gerrit-Branch: master
Gerrit-Change-Id: I3c656b008a3c2b2bb453a59e51d338cb272fa50b
Gerrit-Change-Number: 17125
Gerrit-PatchSet: 1
Gerrit-Owner: pespin <pespin at sysmocom.de>
Gerrit-Reviewer: Jenkins Builder
Gerrit-Reviewer: pespin <pespin at sysmocom.de>
Gerrit-MessageType: merged
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osmocom.org/pipermail/gerrit-log/attachments/20200211/2adb6fb0/attachment.htm>