<p>Pau Espin Pedrol has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.osmocom.org/9299">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">Add option to expect bts/pcu failures and respawn its processes<br><br>Some tests may want to reproduce some scenarios in which it is expected<br>that a BTS process is stopped, for instance if the BSC link is dropped.<br>Provide a keepalive parameter to start() for bts and pcu objects to<br>inform suite that failures are expected and that it should keep them<br>alive in case that ocurrs by respawning the BTS process.<br><br>Change-Id: Ia2a7539f9fad457125ac9b60a52a52999e885ba8<br>---<br>M src/osmo_gsm_tester/bts.py<br>M src/osmo_gsm_tester/bts_nanobts.py<br>M src/osmo_gsm_tester/bts_osmo.py<br>M src/osmo_gsm_tester/bts_osmotrx.py<br>M src/osmo_gsm_tester/bts_sysmo.py<br>M src/osmo_gsm_tester/pcu.py<br>M src/osmo_gsm_tester/pcu_osmo.py<br>M src/osmo_gsm_tester/pcu_sysmo.py<br>M src/osmo_gsm_tester/process.py<br>M src/osmo_gsm_tester/suite.py<br>10 files changed, 57 insertions(+), 38 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/99/9299/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/src/osmo_gsm_tester/bts.py b/src/osmo_gsm_tester/bts.py</span><br><span>index f59cff3..ca33eb4 100644</span><br><span>--- a/src/osmo_gsm_tester/bts.py</span><br><span>+++ b/src/osmo_gsm_tester/bts.py</span><br><span>@@ -139,8 +139,9 @@</span><br><span> # PUBLIC (test API included)</span><br><span> ###################</span><br><span> @abstractmethod</span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(0, 100%, 40%);">- 'Starts BTS proccess and sets self.proc_bts with an object of Process interface'</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''Starts BTS. If keepalive is set, it will expect internal issues and</span><br><span style="color: hsl(120, 100%, 40%);">+ respawn related processes when detected'''</span><br><span> pass</span><br><span> </span><br><span> @abstractmethod</span><br><span>diff --git a/src/osmo_gsm_tester/bts_nanobts.py b/src/osmo_gsm_tester/bts_nanobts.py</span><br><span>index d631f1c..29a8ac5 100644</span><br><span>--- a/src/osmo_gsm_tester/bts_nanobts.py</span><br><span>+++ b/src/osmo_gsm_tester/bts_nanobts.py</span><br><span>@@ -80,7 +80,7 @@</span><br><span> # PUBLIC (test API included)</span><br><span> ###################</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> if self.conf.get('ipa_unit_id') is None:</span><br><span> raise log.Error('No attribute %s provided in conf!' % attr)</span><br><span> self.run_dir = util.Dir(self.suite_run.get_test_run_dir().new_dir(self.name()))</span><br><span>diff --git a/src/osmo_gsm_tester/bts_osmo.py b/src/osmo_gsm_tester/bts_osmo.py</span><br><span>index 21ae135..b9b7fef 100644</span><br><span>--- a/src/osmo_gsm_tester/bts_osmo.py</span><br><span>+++ b/src/osmo_gsm_tester/bts_osmo.py</span><br><span>@@ -57,7 +57,7 @@</span><br><span> # PUBLIC (test API included)</span><br><span> ###################</span><br><span> @abstractmethod</span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> # coming from bts.Bts, we forward the implementation to children.</span><br><span> pass</span><br><span> </span><br><span>@@ -108,6 +108,6 @@</span><br><span> # PUBLIC (test API included)</span><br><span> ###################</span><br><span> @abstractmethod</span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> # coming from bts.Bts, we forward the implementation to children.</span><br><span> pass</span><br><span>diff --git a/src/osmo_gsm_tester/bts_osmotrx.py b/src/osmo_gsm_tester/bts_osmotrx.py</span><br><span>index 9f76194..b9310f8 100644</span><br><span>--- a/src/osmo_gsm_tester/bts_osmotrx.py</span><br><span>+++ b/src/osmo_gsm_tester/bts_osmotrx.py</span><br><span>@@ -51,7 +51,7 @@</span><br><span> def launch_trx_enabled(self):</span><br><span> return util.str2bool(self.conf.get('launch_trx'))</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def launch_process(self, binary_name, *args):</span><br><span style="color: hsl(120, 100%, 40%);">+ def launch_process(self, keepalive, binary_name, *args):</span><br><span> binary = os.path.abspath(self.inst.child('bin', binary_name))</span><br><span> run_dir = self.run_dir.new_dir(binary_name)</span><br><span> if not os.path.isfile(binary):</span><br><span>@@ -59,7 +59,7 @@</span><br><span> proc = process.Process(binary_name, run_dir,</span><br><span> (binary,) + args,</span><br><span> env=self.env)</span><br><span style="color: hsl(0, 100%, 40%);">- self.suite_run.remember_to_stop(proc)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.suite_run.remember_to_stop(proc, keepalive)</span><br><span> proc.launch()</span><br><span> return proc</span><br><span> </span><br><span>@@ -99,7 +99,7 @@</span><br><span> ###################</span><br><span> # PUBLIC (test API included)</span><br><span> ###################</span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> if self.bsc is None:</span><br><span> raise RuntimeError('BTS needs to be added to a BSC or NITB before it can be started')</span><br><span> self.suite_run.poll()</span><br><span>@@ -110,7 +110,7 @@</span><br><span> </span><br><span> if self.launch_trx_enabled():</span><br><span> self.trx = OsmoTrx(self.suite_run, self.conf, self.trx_remote_ip(), self.remote_addr())</span><br><span style="color: hsl(0, 100%, 40%);">- self.trx.start()</span><br><span style="color: hsl(120, 100%, 40%);">+ self.trx.start(keepalive)</span><br><span> self.log('Waiting for osmo-trx to start up...')</span><br><span> MainLoop.wait(self, self.trx.trx_ready)</span><br><span> </span><br><span>@@ -120,7 +120,7 @@</span><br><span> raise RuntimeError('No lib/ in %r' % self.inst)</span><br><span> self.env = { 'LD_LIBRARY_PATH': util.prepend_library_path(lib) }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- self.proc_bts = self.launch_process(OsmoBtsTrx.BIN_BTS_TRX, '-r', '1',</span><br><span style="color: hsl(120, 100%, 40%);">+ self.proc_bts = self.launch_process(keepalive, OsmoBtsTrx.BIN_BTS_TRX, '-r', '1',</span><br><span> '-c', os.path.abspath(self.config_file),</span><br><span> '-i', self.bsc.addr())</span><br><span> self.suite_run.poll()</span><br><span>@@ -163,17 +163,17 @@</span><br><span> self.dbg(r)</span><br><span> f.write(r)</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> self.run_dir = util.Dir(self.suite_run.get_test_run_dir().new_dir(self.name()))</span><br><span> self.configure()</span><br><span> self.inst = util.Dir(os.path.abspath(self.suite_run.trial.get_inst('osmo-trx')))</span><br><span> lib = self.inst.child('lib')</span><br><span> self.env = { 'LD_LIBRARY_PATH': util.prepend_library_path(lib) }</span><br><span style="color: hsl(0, 100%, 40%);">- self.proc_trx = self.launch_process(OsmoTrx.BIN_TRX, '-x',</span><br><span style="color: hsl(120, 100%, 40%);">+ self.proc_trx = self.launch_process(keepalive, OsmoTrx.BIN_TRX, '-x',</span><br><span> '-j', self.listen_ip, '-i', self.bts_ip,</span><br><span> '-C', os.path.abspath(self.config_file))</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def launch_process(self, binary_name, *args):</span><br><span style="color: hsl(120, 100%, 40%);">+ def launch_process(self, keepalive, binary_name, *args):</span><br><span> binary = os.path.abspath(self.inst.child('bin', binary_name))</span><br><span> run_dir = self.run_dir.new_dir(binary_name)</span><br><span> if not os.path.isfile(binary):</span><br><span>@@ -181,7 +181,7 @@</span><br><span> proc = process.Process(binary_name, run_dir,</span><br><span> (binary,) + args,</span><br><span> env=self.env)</span><br><span style="color: hsl(0, 100%, 40%);">- self.suite_run.remember_to_stop(proc)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.suite_run.remember_to_stop(proc, keepalive)</span><br><span> proc.launch()</span><br><span> return proc</span><br><span> </span><br><span>diff --git a/src/osmo_gsm_tester/bts_sysmo.py b/src/osmo_gsm_tester/bts_sysmo.py</span><br><span>index d0f6ff3..65c9279 100644</span><br><span>--- a/src/osmo_gsm_tester/bts_sysmo.py</span><br><span>+++ b/src/osmo_gsm_tester/bts_sysmo.py</span><br><span>@@ -54,9 +54,9 @@</span><br><span> log.ctx(proc)</span><br><span> raise log.Error('Exited in error')</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def launch_remote(self, name, popen_args, remote_cwd=None):</span><br><span style="color: hsl(120, 100%, 40%);">+ def launch_remote(self, name, popen_args, remote_cwd=None, keepalive=False):</span><br><span> proc = self._process_remote(name, popen_args, remote_cwd)</span><br><span style="color: hsl(0, 100%, 40%);">- self.suite_run.remember_to_stop(proc)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.suite_run.remember_to_stop(proc, keepalive)</span><br><span> proc.launch()</span><br><span> return proc</span><br><span> </span><br><span>@@ -110,7 +110,7 @@</span><br><span> ###################</span><br><span> # PUBLIC (test API included)</span><br><span> ###################</span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> if self.bsc is None:</span><br><span> raise RuntimeError('BTS needs to be added to a BSC or NITB before it can be started')</span><br><span> log.log('Starting sysmoBTS to connect to', self.bsc)</span><br><span>@@ -151,6 +151,6 @@</span><br><span> if self._direct_pcu_enabled():</span><br><span> args += ('-M',)</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- self.proc_bts = self.launch_remote('osmo-bts-sysmo', args, remote_cwd=remote_run_dir)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.proc_bts = self.launch_remote('osmo-bts-sysmo', args, remote_cwd=remote_run_dir, keepalive=keepalive)</span><br><span> </span><br><span> # vim: expandtab tabstop=4 shiftwidth=4</span><br><span>diff --git a/src/osmo_gsm_tester/pcu.py b/src/osmo_gsm_tester/pcu.py</span><br><span>index 97d0b92..0260296 100644</span><br><span>--- a/src/osmo_gsm_tester/pcu.py</span><br><span>+++ b/src/osmo_gsm_tester/pcu.py</span><br><span>@@ -42,7 +42,7 @@</span><br><span> ###################</span><br><span> </span><br><span> @abstractmethod</span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> """Start the PCU. Must be implemented by subclass."""</span><br><span> pass</span><br><span> </span><br><span>@@ -54,7 +54,7 @@</span><br><span> def __init__(self, suite_run, bts, conf):</span><br><span> super().__init__(suite_run, bts, conf, 'PcuDummy')</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> pass</span><br><span> </span><br><span> # vim: expandtab tabstop=4 shiftwidth=4</span><br><span>diff --git a/src/osmo_gsm_tester/pcu_osmo.py b/src/osmo_gsm_tester/pcu_osmo.py</span><br><span>index 6ab97de..50ae134 100644</span><br><span>--- a/src/osmo_gsm_tester/pcu_osmo.py</span><br><span>+++ b/src/osmo_gsm_tester/pcu_osmo.py</span><br><span>@@ -34,7 +34,7 @@</span><br><span> self.conf = conf</span><br><span> self.env = {}</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> self.run_dir = util.Dir(self.suite_run.get_test_run_dir().new_dir(self.name()))</span><br><span> self.configure()</span><br><span> </span><br><span>@@ -44,12 +44,12 @@</span><br><span> raise RuntimeError('No lib/ in %r' % self.inst)</span><br><span> self.env = { 'LD_LIBRARY_PATH': util.prepend_library_path(lib) }</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- self.launch_process(OsmoPcu.BIN_PCU, '-r', '1',</span><br><span style="color: hsl(120, 100%, 40%);">+ self.launch_process(keepalive, OsmoPcu.BIN_PCU, '-r', '1',</span><br><span> '-c', os.path.abspath(self.config_file),</span><br><span> '-i', self.bts.bsc.addr())</span><br><span> self.suite_run.poll()</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def launch_process(self, binary_name, *args):</span><br><span style="color: hsl(120, 100%, 40%);">+ def launch_process(self, keepalive, binary_name, *args):</span><br><span> binary = os.path.abspath(self.inst.child('bin', binary_name))</span><br><span> run_dir = self.run_dir.new_dir(binary_name)</span><br><span> if not os.path.isfile(binary):</span><br><span>@@ -57,7 +57,7 @@</span><br><span> proc = process.Process(binary_name, run_dir,</span><br><span> (binary,) + args,</span><br><span> env=self.env)</span><br><span style="color: hsl(0, 100%, 40%);">- self.suite_run.remember_to_stop(proc)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.suite_run.remember_to_stop(proc, keepalive)</span><br><span> proc.launch()</span><br><span> return proc</span><br><span> </span><br><span>diff --git a/src/osmo_gsm_tester/pcu_sysmo.py b/src/osmo_gsm_tester/pcu_sysmo.py</span><br><span>index 675de50..b97852a 100644</span><br><span>--- a/src/osmo_gsm_tester/pcu_sysmo.py</span><br><span>+++ b/src/osmo_gsm_tester/pcu_sysmo.py</span><br><span>@@ -43,7 +43,7 @@</span><br><span> self.remote_env = {}</span><br><span> self.remote_user = 'root'</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def start(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ def start(self, keepalive=False):</span><br><span> self.run_dir = util.Dir(self.suite_run.get_test_run_dir().new_dir(self.name()))</span><br><span> self.configure()</span><br><span> </span><br><span>@@ -75,7 +75,7 @@</span><br><span> ('LD_LIBRARY_PATH=%s' % remote_lib,</span><br><span> remote_binary, '-c', remote_config_file, '-r', '1',</span><br><span> '-i', self.sysmobts.bsc.addr()),</span><br><span style="color: hsl(0, 100%, 40%);">- remote_cwd=remote_run_dir)</span><br><span style="color: hsl(120, 100%, 40%);">+ remote_cwd=remote_run_dir, keepalive=keepalive)</span><br><span> </span><br><span> def _process_remote(self, name, popen_args, remote_cwd=None):</span><br><span> run_dir = self.run_dir.new_dir(name)</span><br><span>@@ -90,9 +90,9 @@</span><br><span> log.ctx(proc)</span><br><span> raise log.Error('Exited in error')</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def launch_remote(self, name, popen_args, remote_cwd=None):</span><br><span style="color: hsl(120, 100%, 40%);">+ def launch_remote(self, name, popen_args, remote_cwd=None, keepalive=False):</span><br><span> proc = self._process_remote(name, popen_args, remote_cwd)</span><br><span style="color: hsl(0, 100%, 40%);">- self.suite_run.remember_to_stop(proc)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.suite_run.remember_to_stop(proc, keepalive)</span><br><span> proc.launch()</span><br><span> </span><br><span> def run_local(self, name, popen_args):</span><br><span>diff --git a/src/osmo_gsm_tester/process.py b/src/osmo_gsm_tester/process.py</span><br><span>index 477a096..c13ded0 100644</span><br><span>--- a/src/osmo_gsm_tester/process.py</span><br><span>+++ b/src/osmo_gsm_tester/process.py</span><br><span>@@ -79,6 +79,13 @@</span><br><span> self.set_name(self.name_str, pid=self.process_obj.pid)</span><br><span> self.log('Launched')</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def respawn(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.dbg('respawn')</span><br><span style="color: hsl(120, 100%, 40%);">+ assert not self.is_running()</span><br><span style="color: hsl(120, 100%, 40%);">+ self.result = None</span><br><span style="color: hsl(120, 100%, 40%);">+ self.killed = None</span><br><span style="color: hsl(120, 100%, 40%);">+ self.launch()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> def _poll_termination(self, time_to_wait_for_term=5):</span><br><span> wait_step = 0.001</span><br><span> waited_time = 0</span><br><span>diff --git a/src/osmo_gsm_tester/suite.py b/src/osmo_gsm_tester/suite.py</span><br><span>index 76cd248..618a39b 100644</span><br><span>--- a/src/osmo_gsm_tester/suite.py</span><br><span>+++ b/src/osmo_gsm_tester/suite.py</span><br><span>@@ -230,19 +230,27 @@</span><br><span> skipped += 1</span><br><span> return (passed, skipped, failed)</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- def remember_to_stop(self, process):</span><br><span style="color: hsl(120, 100%, 40%);">+ def remember_to_stop(self, process, respawn=False):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''Ask suite to monitor and manage lifecycle of the Process object. If a</span><br><span style="color: hsl(120, 100%, 40%);">+ process managed by suite finishes before cleanup time, the current test</span><br><span style="color: hsl(120, 100%, 40%);">+ will be marked as FAIL and end immediatelly. If respwan=True, then suite</span><br><span style="color: hsl(120, 100%, 40%);">+ will respawn() the process instead.'''</span><br><span> if self._processes is None:</span><br><span> self._processes = []</span><br><span style="color: hsl(0, 100%, 40%);">- self._processes.insert(0, process)</span><br><span style="color: hsl(120, 100%, 40%);">+ self._processes.insert(0, (process, respawn))</span><br><span> </span><br><span> def stop_processes(self):</span><br><span> while self._processes:</span><br><span style="color: hsl(0, 100%, 40%);">- self._processes.pop().terminate()</span><br><span style="color: hsl(120, 100%, 40%);">+ process, respawn = self._processes.pop()</span><br><span style="color: hsl(120, 100%, 40%);">+ process.terminate()</span><br><span> </span><br><span> def stop_process(self, process):</span><br><span> 'Remove process from monitored list and stop it'</span><br><span style="color: hsl(0, 100%, 40%);">- self._processes.remove(process)</span><br><span style="color: hsl(0, 100%, 40%);">- process.terminate()</span><br><span style="color: hsl(120, 100%, 40%);">+ for proc_respawn in self._processes:</span><br><span style="color: hsl(120, 100%, 40%);">+ proc, respawn = proc_respawn</span><br><span style="color: hsl(120, 100%, 40%);">+ if proc == process:</span><br><span style="color: hsl(120, 100%, 40%);">+ self._processes.remove(proc_respawn)</span><br><span style="color: hsl(120, 100%, 40%);">+ proc.terminate()</span><br><span> </span><br><span> def free_resources(self):</span><br><span> if self.reserved_resources is None:</span><br><span>@@ -351,12 +359,15 @@</span><br><span> </span><br><span> def poll(self):</span><br><span> if self._processes:</span><br><span style="color: hsl(0, 100%, 40%);">- for process in self._processes:</span><br><span style="color: hsl(120, 100%, 40%);">+ for process, respawn in self._processes:</span><br><span> if process.terminated():</span><br><span style="color: hsl(0, 100%, 40%);">- process.log_stdout_tail()</span><br><span style="color: hsl(0, 100%, 40%);">- process.log_stderr_tail()</span><br><span style="color: hsl(0, 100%, 40%);">- log.ctx(process)</span><br><span style="color: hsl(0, 100%, 40%);">- raise log.Error('Process ended prematurely: %s' % process.name())</span><br><span style="color: hsl(120, 100%, 40%);">+ if respawn == True:</span><br><span style="color: hsl(120, 100%, 40%);">+ process.respawn()</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ process.log_stdout_tail()</span><br><span style="color: hsl(120, 100%, 40%);">+ process.log_stderr_tail()</span><br><span style="color: hsl(120, 100%, 40%);">+ log.ctx(process)</span><br><span style="color: hsl(120, 100%, 40%);">+ raise log.Error('Process ended prematurely: %s' % process.name())</span><br><span> </span><br><span> def prompt(self, *msgs, **msg_details):</span><br><span> 'ask for user interaction. Do not use in tests that should run automatically!'</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/9299">change 9299</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/9299"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>
<div style="display:none"> Gerrit-Change-Id: Ia2a7539f9fad457125ac9b60a52a52999e885ba8 </div>
<div style="display:none"> Gerrit-Change-Number: 9299 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: Pau Espin Pedrol <pespin@sysmocom.de> </div>