<p>Holger Freyther <strong>merged</strong> this change.</p><p><a href="https://gerrit.osmocom.org/13087">View Change</a></p><div style="white-space:pre-wrap">Approvals:
Holger Freyther: Looks good to me, approved
Jenkins Builder: Verified
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">process: Make killing processes non-sequential<br><br>Change-Id: Icf1ac6774ea11880542012fd6c6ac73302bb74f5<br>---<br>M selftest/suite_test.ok<br>M src/osmo_gsm_tester/process.py<br>2 files changed, 129 insertions(+), 3 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok</span><br><span>index 908f24f..85e94b2 100644</span><br><span>--- a/selftest/suite_test.ok</span><br><span>+++ b/selftest/suite_test.ok</span><br><span>@@ -96,6 +96,14 @@</span><br><span> tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]]</span><br><span> tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]]</span><br><span> tst hello_world.py:[LINENR] Test passed (N.N sec) [test_suite↪hello_world.py]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes.</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes.</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite PASS</span><br><span> ---------------------------------------------------------------------</span><br><span>@@ -119,6 +127,14 @@</span><br><span> tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]]</span><br><span> tst test_error.py:[LINENR]: ERR: AssertionError: test_error.py:[LINENR]: assert False [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False]</span><br><span> tst test_error.py:[LINENR]: Test FAILED (N.N sec) [test_suite↪test_error.py:[LINENR]] [test.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite FAIL</span><br><span> ---------------------------------------------------------------------</span><br><span>@@ -142,6 +158,14 @@</span><br><span> tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]</span><br><span> tst test_fail.py:[LINENR]: ERR: EpicFail: This failure is expected [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]</span><br><span> tst test_fail.py:[LINENR]: Test FAILED (N.N sec) [test_suite↪test_fail.py:[LINENR]] [test.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite FAIL</span><br><span> ---------------------------------------------------------------------</span><br><span>@@ -164,6 +188,14 @@</span><br><span> ----------------------------------------------</span><br><span> tst test_fail_raise.py:[LINENR]: ERR: ExpectedFail: This failure is expected [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]: raise ExpectedFail('This failure is expected')]</span><br><span> tst test_fail_raise.py:[LINENR]: Test FAILED (N.N sec) [test_suite↪test_fail_raise.py:[LINENR]] [test.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite FAIL</span><br><span> ---------------------------------------------------------------------</span><br><span>@@ -251,6 +283,14 @@</span><br><span> tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]</span><br><span> tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]</span><br><span> tst hello_world.py:[LINENR] Test passed (N.N sec) [test_suite↪hello_world.py] [test.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite PASS</span><br><span> ---------------------------------------------------------------------</span><br><span>@@ -338,6 +378,14 @@</span><br><span> tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]</span><br><span> tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]</span><br><span> tst hello_world.py:[LINENR] Test passed (N.N sec) [test_suite↪hello_world.py] [test.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite PASS</span><br><span> ---------------------------------------------------------------------</span><br><span>@@ -471,6 +519,14 @@</span><br><span> tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]</span><br><span> tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] [hello_world.py:[LINENR]]</span><br><span> tst hello_world.py:[LINENR] Test passed (N.N sec) [test_suite↪hello_world.py] [test.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Scheduled to terminate 0 processes. [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGTERM [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGINT [process.py:[LINENR]]</span><br><span style="color: hsl(120, 100%, 40%);">+--- ParallelTerminationStrategy: DBG: Starting to kill with SIGKILL [process.py:[LINENR]]</span><br><span> ---------------------------------------------------------------------</span><br><span> trial test_suite PASS</span><br><span> ---------------------------------------------------------------------</span><br><span>diff --git a/src/osmo_gsm_tester/process.py b/src/osmo_gsm_tester/process.py</span><br><span>index 3050f83..a104e10 100644</span><br><span>--- a/src/osmo_gsm_tester/process.py</span><br><span>+++ b/src/osmo_gsm_tester/process.py</span><br><span>@@ -47,10 +47,75 @@</span><br><span> class ParallelTerminationStrategy(TerminationStrategy):</span><br><span> """Processes will be terminated in parallel."""</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def terminate_all(self):</span><br><span style="color: hsl(0, 100%, 40%);">- # TODO(zecke): Actually make this non-sequential.</span><br><span style="color: hsl(120, 100%, 40%);">+ def _prune_dead_processes(self, poll_first):</span><br><span style="color: hsl(120, 100%, 40%);">+ """Removes all dead processes from the list."""</span><br><span style="color: hsl(120, 100%, 40%);">+ # Remove all processes that terminated!</span><br><span style="color: hsl(120, 100%, 40%);">+ self._processes = list(filter(lambda proc: proc.is_running(poll_first), self._processes))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def _build_process_map(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ """Builds a mapping from pid to process."""</span><br><span style="color: hsl(120, 100%, 40%);">+ self._process_map = {}</span><br><span> for process in self._processes:</span><br><span style="color: hsl(0, 100%, 40%);">- process.terminate()</span><br><span style="color: hsl(120, 100%, 40%);">+ pid = process.pid()</span><br><span style="color: hsl(120, 100%, 40%);">+ if pid is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ self._process_map[pid] = process</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def _poll_once(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ """Polls for to be collected children once."""</span><br><span style="color: hsl(120, 100%, 40%);">+ pid, result = os.waitpid(0, os.WNOHANG)</span><br><span style="color: hsl(120, 100%, 40%);">+ # Did some other process die?</span><br><span style="color: hsl(120, 100%, 40%);">+ if pid == 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ return False</span><br><span style="color: hsl(120, 100%, 40%);">+ proc = self._process_map.get(pid)</span><br><span style="color: hsl(120, 100%, 40%);">+ if proc is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.dbg("Unknown process with pid(%d) died." % pid)</span><br><span style="color: hsl(120, 100%, 40%);">+ return False</span><br><span style="color: hsl(120, 100%, 40%);">+ # Update the process state and forget about it</span><br><span style="color: hsl(120, 100%, 40%);">+ self.log("PID %d died..." % pid)</span><br><span style="color: hsl(120, 100%, 40%);">+ proc.result = result</span><br><span style="color: hsl(120, 100%, 40%);">+ proc.cleanup()</span><br><span style="color: hsl(120, 100%, 40%);">+ self._processes.remove(proc)</span><br><span style="color: hsl(120, 100%, 40%);">+ del self._process_map[pid]</span><br><span style="color: hsl(120, 100%, 40%);">+ return True</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def _poll_for_termination(self, time_to_wait_for_term=5):</span><br><span style="color: hsl(120, 100%, 40%);">+ """Waits for the termination of processes until timeout|all ended."""</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ wait_step = 0.001</span><br><span style="color: hsl(120, 100%, 40%);">+ waited_time = 0</span><br><span style="color: hsl(120, 100%, 40%);">+ while len(self._processes) > 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ # Collect processes until there are none to be collected.</span><br><span style="color: hsl(120, 100%, 40%);">+ while True:</span><br><span style="color: hsl(120, 100%, 40%);">+ try:</span><br><span style="color: hsl(120, 100%, 40%);">+ if not self._poll_once():</span><br><span style="color: hsl(120, 100%, 40%);">+ break</span><br><span style="color: hsl(120, 100%, 40%);">+ except ChildProcessError:</span><br><span style="color: hsl(120, 100%, 40%);">+ break</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # All processes died and we can return before sleeping</span><br><span style="color: hsl(120, 100%, 40%);">+ if len(self._processes) == 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ break</span><br><span style="color: hsl(120, 100%, 40%);">+ waited_time += wait_step</span><br><span style="color: hsl(120, 100%, 40%);">+ # make wait_step approach 1.0</span><br><span style="color: hsl(120, 100%, 40%);">+ wait_step = (1. + 5. * wait_step) / 6.</span><br><span style="color: hsl(120, 100%, 40%);">+ if waited_time >= time_to_wait_for_term:</span><br><span style="color: hsl(120, 100%, 40%);">+ break</span><br><span style="color: hsl(120, 100%, 40%);">+ time.sleep(wait_step)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def terminate_all(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.dbg("Scheduled to terminate %d processes." % len(self._processes))</span><br><span style="color: hsl(120, 100%, 40%);">+ self._prune_dead_processes(True)</span><br><span style="color: hsl(120, 100%, 40%);">+ self._build_process_map()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Iterate through all signals.</span><br><span style="color: hsl(120, 100%, 40%);">+ for sig in [signal.SIGTERM, signal.SIGINT, signal.SIGKILL]:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.dbg("Starting to kill with %s" % sig.name)</span><br><span style="color: hsl(120, 100%, 40%);">+ for process in self._processes:</span><br><span style="color: hsl(120, 100%, 40%);">+ process.kill(sig)</span><br><span style="color: hsl(120, 100%, 40%);">+ if sig == signal.SIGKILL:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ self._poll_for_termination()</span><br><span> </span><br><span> </span><br><span> class Process(log.Origin):</span><br><span>@@ -145,6 +210,11 @@</span><br><span> def send_signal(self, sig):</span><br><span> os.kill(self.process_obj.pid, sig)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def pid(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ if self.process_obj is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ return None</span><br><span style="color: hsl(120, 100%, 40%);">+ return self.process_obj.pid</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> def kill(self, sig):</span><br><span> """Kills the process with the given signal and remembers it."""</span><br><span> self.log('Terminating (%s)' % sig.name)</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/13087">change 13087</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/13087"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: merged </div>
<div style="display:none"> Gerrit-Change-Id: Icf1ac6774ea11880542012fd6c6ac73302bb74f5 </div>
<div style="display:none"> Gerrit-Change-Number: 13087 </div>
<div style="display:none"> Gerrit-PatchSet: 4 </div>
<div style="display:none"> Gerrit-Owner: Holger Freyther <holger@freyther.de> </div>
<div style="display:none"> Gerrit-Reviewer: Harald Welte <laforge@gnumonks.org> </div>
<div style="display:none"> Gerrit-Reviewer: Holger Freyther <holger@freyther.de> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder (1000002) </div>
<div style="display:none"> Gerrit-Reviewer: Pau Espin Pedrol <pespin@sysmocom.de> </div>