<p>Holger Freyther <strong>merged</strong> this change.</p><p><a href="https://gerrit.osmocom.org/13130">View Change</a></p><div style="white-space:pre-wrap">Approvals:
Pau Espin Pedrol: Looks good to me, approved
Jenkins Builder: Verified
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">nitb_netreg_mass: Add code to declare a run successful/failure<br><br>Change-Id: If26047f0635db8c7209af143fbd1da4c9b2312d1<br>---<br>M src/osmo_gsm_tester/ms_driver.py<br>M src/osmo_ms_driver/location_update_test.py<br>M suites/nitb_netreg_mass/register_default_mass.py<br>3 files changed, 71 insertions(+), 7 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/src/osmo_gsm_tester/ms_driver.py b/src/osmo_gsm_tester/ms_driver.py</span><br><span>index 8ee93b7..96b907a 100644</span><br><span>--- a/src/osmo_gsm_tester/ms_driver.py</span><br><span>+++ b/src/osmo_gsm_tester/ms_driver.py</span><br><span>@@ -118,6 +118,18 @@</span><br><span> """</span><br><span> self._test_case.print_stats()</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def get_stats(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+ Returns a statistical summary of the test.</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+ return self._test_case.get_stats()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def get_result_values(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+ Returns the raw result values of the test run in any order.</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+ return self._test_case.get_result_values()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> def cleanup(self):</span><br><span> """</span><br><span> Cleans up the driver (e.g. AF_UNIX files).</span><br><span>diff --git a/src/osmo_ms_driver/location_update_test.py b/src/osmo_ms_driver/location_update_test.py</span><br><span>index 2e3a648..5ff2199 100644</span><br><span>--- a/src/osmo_ms_driver/location_update_test.py</span><br><span>+++ b/src/osmo_ms_driver/location_update_test.py</span><br><span>@@ -23,9 +23,11 @@</span><br><span> </span><br><span> from datetime import timedelta</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+import collections</span><br><span> import time</span><br><span> </span><br><span> class LUResult(Results):</span><br><span style="color: hsl(120, 100%, 40%);">+ """Representation of a Location Updating Result."""</span><br><span> </span><br><span> def __init__(self, name):</span><br><span> super().__init__(name)</span><br><span>@@ -44,6 +46,10 @@</span><br><span> def lu_delay(self):</span><br><span> return self.lu_time() - self.start_time()</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+LUStats = collections.namedtuple("LUStats", ["num_attempted", "num_completed",</span><br><span style="color: hsl(120, 100%, 40%);">+ "min_latency", "max_latency"])</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> class MassUpdateLocationTest(log.Origin):</span><br><span> """</span><br><span> A test to launch a configurable amount of MS and make them</span><br><span>@@ -219,9 +225,24 @@</span><br><span> max_value = result.lu_delay()</span><br><span> return min_value, max_value</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def get_result_values(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+ Returns the raw result values of the test run in any order.</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+ return self._results.values()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def get_stats(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+ Returns a statistical summary of the test.</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+ attempted = len(self._subscribers)</span><br><span style="color: hsl(120, 100%, 40%);">+ completed = attempted - self._outstanding</span><br><span style="color: hsl(120, 100%, 40%);">+ min_latency, max_latency = self.find_min_max(filter(lambda x: x.has_lu_time(), self._results.values()))</span><br><span style="color: hsl(120, 100%, 40%);">+ return LUStats(attempted, completed, min_latency, max_latency)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> def print_stats(self):</span><br><span style="color: hsl(0, 100%, 40%);">- all_completed = self.all_completed()</span><br><span style="color: hsl(0, 100%, 40%);">- min_value, max_value = self.find_min_max(filter(lambda x: x.has_lu_time(), self._results.values()))</span><br><span style="color: hsl(120, 100%, 40%);">+ stats = self.get_stats()</span><br><span style="color: hsl(120, 100%, 40%);">+ all_completed = stats.num_attempted == stats.num_completed</span><br><span> </span><br><span> self.log("Tests done", all_completed=all_completed,</span><br><span style="color: hsl(0, 100%, 40%);">- min=min_value, max=max_value)</span><br><span style="color: hsl(120, 100%, 40%);">+ min=stats.min_latency, max=stats.max_latency)</span><br><span>diff --git a/suites/nitb_netreg_mass/register_default_mass.py b/suites/nitb_netreg_mass/register_default_mass.py</span><br><span>index e7c91da..76c53f1 100644</span><br><span>--- a/suites/nitb_netreg_mass/register_default_mass.py</span><br><span>+++ b/suites/nitb_netreg_mass/register_default_mass.py</span><br><span>@@ -1,17 +1,18 @@</span><br><span> #!/usr/bin/env python3</span><br><span> """</span><br><span style="color: hsl(0, 100%, 40%);">-Run a network registration with a 'massive' amount of MS</span><br><span style="color: hsl(120, 100%, 40%);">+Runs a network registration with a 'massive' amount of MS</span><br><span> using the ms_driver infrastructure.</span><br><span> """</span><br><span> from osmo_gsm_tester.testenv import *</span><br><span style="color: hsl(120, 100%, 40%);">+from datetime import timedelta</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-print('use resources...')</span><br><span style="color: hsl(120, 100%, 40%);">+print('Claiming resources for the test')</span><br><span> nitb = suite.nitb()</span><br><span> bts = suite.bts()</span><br><span> ms_driver = suite.ms_driver()</span><br><span> modems = suite.all_resources(suite.modem)</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-print('start nitb and bts...')</span><br><span style="color: hsl(120, 100%, 40%);">+print('Launching a simple network')</span><br><span> nitb.bts_add(bts)</span><br><span> nitb.start()</span><br><span> bts.start()</span><br><span>@@ -25,5 +26,35 @@</span><br><span> # Run the base test.</span><br><span> ms_driver.run_test()</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-# Print stats</span><br><span style="color: hsl(120, 100%, 40%);">+# Print the stats of the run.</span><br><span> ms_driver.print_stats()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Evaluate if this run was successful or not. Our initial acceptance criteria</span><br><span style="color: hsl(120, 100%, 40%);">+# is quite basic but it should allow us to scale to a larger number of MS and</span><br><span style="color: hsl(120, 100%, 40%);">+# reasons (e.g. have a full BCCH).</span><br><span style="color: hsl(120, 100%, 40%);">+#</span><br><span style="color: hsl(120, 100%, 40%);">+# 99% of LUs should complete</span><br><span style="color: hsl(120, 100%, 40%);">+# 99% of successful LUs should complete within 10s.</span><br><span style="color: hsl(120, 100%, 40%);">+stats = ms_driver.get_stats()</span><br><span style="color: hsl(120, 100%, 40%);">+if len(modems) > 0 and stats.num_completed < 1:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception("No run completed.")</span><br><span style="color: hsl(120, 100%, 40%);">+completion_ratio = stats.num_attempted / stats.num_completed</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Verify that 99% of LUs completed.</span><br><span style="color: hsl(120, 100%, 40%);">+if completion_ratio < 0.99:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception("Completion ratio of %f%% lower than threshold." % (completion_ratio * 100.0))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+# Check how many results are below our threshold.</span><br><span style="color: hsl(120, 100%, 40%);">+acceptable_delay = timedelta(seconds=30)</span><br><span style="color: hsl(120, 100%, 40%);">+results = ms_driver.get_result_values()</span><br><span style="color: hsl(120, 100%, 40%);">+quick_enough = 0</span><br><span style="color: hsl(120, 100%, 40%);">+for result in results:</span><br><span style="color: hsl(120, 100%, 40%);">+ if not result.has_lu_time():</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ if timedelta(seconds=result.lu_delay()) >= acceptable_delay:</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ quick_enough = quick_enough + 1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+latency_ratio = quick_enough / len(results)</span><br><span style="color: hsl(120, 100%, 40%);">+if latency_ratio < 0.99:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception("Latency ratio of %f%% lower than threshold." % (latency_ratio * 100.0))</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/13130">change 13130</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/13130"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: merged </div>
<div style="display:none"> Gerrit-Change-Id: If26047f0635db8c7209af143fbd1da4c9b2312d1 </div>
<div style="display:none"> Gerrit-Change-Number: 13130 </div>
<div style="display:none"> Gerrit-PatchSet: 3 </div>
<div style="display:none"> Gerrit-Owner: Holger Freyther <holger@freyther.de> </div>
<div style="display:none"> Gerrit-Reviewer: Holger Freyther <holger@freyther.de> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder (1000002) </div>
<div style="display:none"> Gerrit-Reviewer: Pau Espin Pedrol <pespin@sysmocom.de> </div>