<p>pespin <strong>submitted</strong> this change.</p><p><a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/17323">View Change</a></p><div style="white-space:pre-wrap">Approvals:
laforge: Looks good to me, but someone else must approve
pespin: Looks good to me, approved
Jenkins Builder: Verified
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">srsue: Introduce metrics verification procedures<br><br>Change-Id: Ib1da58615cdc4f53ac1a27080e94e5b47760c508<br>---<br>M check_dependencies.py<br>M src/osmo_gsm_tester/srs_enb.py<br>M src/osmo_gsm_tester/srs_ue.py<br>M suites/4g/iperf3.py<br>M suites/4g/suite.conf<br>5 files changed, 122 insertions(+), 7 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/check_dependencies.py b/check_dependencies.py</span><br><span>index 28bfdf7..c3b1d64 100755</span><br><span>--- a/check_dependencies.py</span><br><span>+++ b/check_dependencies.py</span><br><span>@@ -28,5 +28,6 @@</span><br><span> import smpplib</span><br><span> import urllib.request</span><br><span> import xml.etree.ElementTree</span><br><span style="color: hsl(120, 100%, 40%);">+import numpy</span><br><span> </span><br><span> print('dependencies ok')</span><br><span>diff --git a/src/osmo_gsm_tester/srs_enb.py b/src/osmo_gsm_tester/srs_enb.py</span><br><span>index 1cfd212..1fb2db1 100644</span><br><span>--- a/src/osmo_gsm_tester/srs_enb.py</span><br><span>+++ b/src/osmo_gsm_tester/srs_enb.py</span><br><span>@@ -76,6 +76,7 @@</span><br><span> self.remote_config_drb_file = None</span><br><span> self.remote_log_file = None</span><br><span> self._num_prb = 0</span><br><span style="color: hsl(120, 100%, 40%);">+ self._txmode = 0</span><br><span> self.suite_run = suite_run</span><br><span> self.remote_user = conf.get('remote_user', None)</span><br><span> if not rf_type_valid(conf.get('rf_dev_type', None)):</span><br><span>@@ -179,10 +180,13 @@</span><br><span> config.overlay(values, dict(enb=self._conf))</span><br><span> config.overlay(values, dict(enb={ 'mme_addr': self.epc.addr() }))</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ self._num_prb = int(values['enb'].get('num_prb', None))</span><br><span style="color: hsl(120, 100%, 40%);">+ assert self._num_prb</span><br><span style="color: hsl(120, 100%, 40%);">+ self._txmode = int(values['enb'].get('transmission_mode', None))</span><br><span style="color: hsl(120, 100%, 40%);">+ assert self._txmode</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> # We need to set some specific variables programatically here to match IP addresses:</span><br><span> if self._conf.get('rf_dev_type') == 'zmq':</span><br><span style="color: hsl(0, 100%, 40%);">- self._num_prb = int(values['enb'].get('num_prb', None))</span><br><span style="color: hsl(0, 100%, 40%);">- assert self._num_prb</span><br><span> base_srate = num_prb2base_srate(self._num_prb)</span><br><span> rf_dev_args = 'fail_on_disconnect=true,tx_port=tcp://' + self.addr() \</span><br><span> + ':2000,rx_port=tcp://' + self.ue.addr() \</span><br><span>@@ -222,4 +226,31 @@</span><br><span> def num_prb(self):</span><br><span> return self._num_prb</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def ue_max_rate(self, downlink=True):</span><br><span style="color: hsl(120, 100%, 40%);">+ # The max rate for a single UE per PRB in TM1</span><br><span style="color: hsl(120, 100%, 40%);">+ max_phy_rate_tm1_dl = { 6 : 2.3e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 15 : 8e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 25 : 16e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 50 : 36e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 75 : 54e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 100 : 75e6 }</span><br><span style="color: hsl(120, 100%, 40%);">+ # TODO: proper values for this table:</span><br><span style="color: hsl(120, 100%, 40%);">+ max_phy_rate_tm1_ul = { 6 : 0.23e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 15 : 0.8e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 25 : 1.6e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 50 : 3.6e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 75 : 5.4e6,</span><br><span style="color: hsl(120, 100%, 40%);">+ 100 : 7.5e6 }</span><br><span style="color: hsl(120, 100%, 40%);">+ if downlink:</span><br><span style="color: hsl(120, 100%, 40%);">+ max_rate = max_phy_rate_tm1_dl[self.num_prb()]</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ max_rate = max_phy_rate_tm1_ul[self.num_prb()]</span><br><span style="color: hsl(120, 100%, 40%);">+ #TODO: calculate for non-standard prb numbers.</span><br><span style="color: hsl(120, 100%, 40%);">+ if self._txmode > 2:</span><br><span style="color: hsl(120, 100%, 40%);">+ max_rate *= 2</span><br><span style="color: hsl(120, 100%, 40%);">+ # We use 3 control symbols for 6, 15 and 25 PRBs which results in lower max rate</span><br><span style="color: hsl(120, 100%, 40%);">+ if self.num_prb() < 50:</span><br><span style="color: hsl(120, 100%, 40%);">+ max_rate *= 0.9</span><br><span style="color: hsl(120, 100%, 40%);">+ return max_rate</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> # vim: expandtab tabstop=4 shiftwidth=4</span><br><span>diff --git a/src/osmo_gsm_tester/srs_ue.py b/src/osmo_gsm_tester/srs_ue.py</span><br><span>index cbd8a68..f90ea32 100644</span><br><span>--- a/src/osmo_gsm_tester/srs_ue.py</span><br><span>+++ b/src/osmo_gsm_tester/srs_ue.py</span><br><span>@@ -22,6 +22,7 @@</span><br><span> </span><br><span> from . import log, util, config, template, process, remote</span><br><span> from .run_node import RunNode</span><br><span style="color: hsl(120, 100%, 40%);">+from .event_loop import MainLoop</span><br><span> from .ms import MS</span><br><span> </span><br><span> def rf_type_valid(rf_type_str):</span><br><span>@@ -91,10 +92,6 @@</span><br><span> self.rem_host.scpfrom('scp-back-pcap', self.remote_pcap_file, self.pcap_file)</span><br><span> except Exception as e:</span><br><span> self.log(repr(e))</span><br><span style="color: hsl(0, 100%, 40%);">- try:</span><br><span style="color: hsl(0, 100%, 40%);">- self.rem_host.scpfrom('scp-back-metrics', self.remote_metrics_file, self.metrics_file)</span><br><span style="color: hsl(0, 100%, 40%);">- except Exception as e:</span><br><span style="color: hsl(0, 100%, 40%);">- self.log(repr(e))</span><br><span> </span><br><span> def setup_runs_locally(self):</span><br><span> return self.remote_user is None</span><br><span>@@ -102,6 +99,9 @@</span><br><span> def netns(self):</span><br><span> return "srsue1"</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def stop(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ self.suite_run.stop_process(self.process)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> def connect(self, enb):</span><br><span> self.log('Starting srsue')</span><br><span> self.enb = enb</span><br><span>@@ -247,4 +247,80 @@</span><br><span> proc = self.rem_host.RemoteNetNSProcess(name, self.netns(), popen_args, env={})</span><br><span> proc.launch_sync()</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def verify_metric(self, value, operation='avg', metric='dl_brate', criterion='gt'):</span><br><span style="color: hsl(120, 100%, 40%);">+ # file is not properly flushed until the process has stopped.</span><br><span style="color: hsl(120, 100%, 40%);">+ if self.running():</span><br><span style="color: hsl(120, 100%, 40%);">+ self.stop()</span><br><span style="color: hsl(120, 100%, 40%);">+ # metrics file is not flushed immediatelly by the OS during process</span><br><span style="color: hsl(120, 100%, 40%);">+ # tear down, we need to wait some extra time:</span><br><span style="color: hsl(120, 100%, 40%);">+ MainLoop.sleep(self, 2)</span><br><span style="color: hsl(120, 100%, 40%);">+ if not self.setup_runs_locally():</span><br><span style="color: hsl(120, 100%, 40%);">+ try:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.rem_host.scpfrom('scp-back-metrics', self.remote_metrics_file, self.metrics_file)</span><br><span style="color: hsl(120, 100%, 40%);">+ except Exception as e:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.err('Failed copying back metrics file from remote host')</span><br><span style="color: hsl(120, 100%, 40%);">+ raise e</span><br><span style="color: hsl(120, 100%, 40%);">+ metrics = srsUEMetrics(self.metrics_file)</span><br><span style="color: hsl(120, 100%, 40%);">+ return metrics.verify(value, operation, metric, criterion)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+import numpy</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+class srsUEMetrics(log.Origin):</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ VALID_OPERATIONS = ['avg', 'sum']</span><br><span style="color: hsl(120, 100%, 40%);">+ VALID_CRITERION = ['eq','gt','lt']</span><br><span style="color: hsl(120, 100%, 40%);">+ CRITERION_TO_SYM = { 'eq' : '==', 'gt' : '>', 'lt' : '<' }</span><br><span style="color: hsl(120, 100%, 40%);">+ CRYTERION_TO_SYM_OPPOSITE = { 'eq' : '!=', 'gt' : '<=', 'lt' : '>=' }</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __init__(self, metrics_file):</span><br><span style="color: hsl(120, 100%, 40%);">+ super().__init__(log.C_RUN, 'srsue_metrics')</span><br><span style="color: hsl(120, 100%, 40%);">+ self.raw_data = None</span><br><span style="color: hsl(120, 100%, 40%);">+ self.metrics_file = metrics_file</span><br><span style="color: hsl(120, 100%, 40%);">+ # read CSV, guessing data type with first row being the legend</span><br><span style="color: hsl(120, 100%, 40%);">+ try:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.raw_data = numpy.genfromtxt(self.metrics_file, names=True, delimiter=';', dtype=None)</span><br><span style="color: hsl(120, 100%, 40%);">+ except (ValueError, IndexError, IOError) as error:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.err("Error parsing metrics CSV file %s" % self.metrics_file)</span><br><span style="color: hsl(120, 100%, 40%);">+ raise error</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def verify(self, value, operation='avg', metric='dl_brate', criterion='gt'):</span><br><span style="color: hsl(120, 100%, 40%);">+ if operation not in self.VALID_OPERATIONS:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise log.Error('Unknown operation %s not in %r' % (operation, self.VALID_OPERATIONS))</span><br><span style="color: hsl(120, 100%, 40%);">+ if criterion not in self.VALID_CRITERION:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise log.Error('Unknown operation %s not in %r' % (operation, self.VALID_CRITERION))</span><br><span style="color: hsl(120, 100%, 40%);">+ # check if given metric exists in data</span><br><span style="color: hsl(120, 100%, 40%);">+ try:</span><br><span style="color: hsl(120, 100%, 40%);">+ sel_data = self.raw_data[metric]</span><br><span style="color: hsl(120, 100%, 40%);">+ except ValueError as err:</span><br><span style="color: hsl(120, 100%, 40%);">+ print('metric %s not available' % metric)</span><br><span style="color: hsl(120, 100%, 40%);">+ raise err</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if operation == 'avg':</span><br><span style="color: hsl(120, 100%, 40%);">+ result = numpy.average(sel_data)</span><br><span style="color: hsl(120, 100%, 40%);">+ elif operation == 'sum':</span><br><span style="color: hsl(120, 100%, 40%);">+ result = numpy.sum(sel_data)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.dbg(result=result, value=value)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ success = False</span><br><span style="color: hsl(120, 100%, 40%);">+ if criterion == 'eq' and result == value or \</span><br><span style="color: hsl(120, 100%, 40%);">+ criterion == 'gt' and result > value or \</span><br><span style="color: hsl(120, 100%, 40%);">+ criterion == 'lt' and result < value:</span><br><span style="color: hsl(120, 100%, 40%);">+ success = True</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Convert bitrate in Mbit/s:</span><br><span style="color: hsl(120, 100%, 40%);">+ if metric.find('brate') > 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ result /= 1e6</span><br><span style="color: hsl(120, 100%, 40%);">+ value /= 1e6</span><br><span style="color: hsl(120, 100%, 40%);">+ mbit_str = ' Mbit/s'</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ mbit_str = ''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if not success:</span><br><span style="color: hsl(120, 100%, 40%);">+ result_msg = "{:.2f}{} {} {:.2f}{}".format(result, mbit_str, self.CRYTERION_TO_SYM_OPPOSITE[criterion], value, mbit_str)</span><br><span style="color: hsl(120, 100%, 40%);">+ raise log.Error(result_msg)</span><br><span style="color: hsl(120, 100%, 40%);">+ result_msg = "{:.2f}{} {} {:.2f}{}".format(result, mbit_str, self.CRITERION_TO_SYM[criterion], value, mbit_str)</span><br><span style="color: hsl(120, 100%, 40%);">+ # TODO: overwrite test system-out with this text.</span><br><span style="color: hsl(120, 100%, 40%);">+ return result_msg</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> # vim: expandtab tabstop=4 shiftwidth=4</span><br><span>diff --git a/suites/4g/iperf3.py b/suites/4g/iperf3.py</span><br><span>index 371376f..20489b5 100755</span><br><span>--- a/suites/4g/iperf3.py</span><br><span>+++ b/suites/4g/iperf3.py</span><br><span>@@ -32,7 +32,7 @@</span><br><span> ue.connect(enb)</span><br><span> </span><br><span> iperf3srv.start()</span><br><span style="color: hsl(0, 100%, 40%);">-proc = iperf3cli.prepare_test_proc(False, ue.netns())</span><br><span style="color: hsl(120, 100%, 40%);">+proc = iperf3cli.prepare_test_proc(False, ue.netns(), time_sec=60)</span><br><span> </span><br><span> print('waiting for UE to attach...')</span><br><span> wait(ue.is_connected, None)</span><br><span>@@ -42,3 +42,7 @@</span><br><span> proc.launch_sync()</span><br><span> iperf3srv.stop()</span><br><span> print_results(iperf3cli.get_results(), iperf3srv.get_results())</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+max_rate = enb.ue_max_rate(downlink=False)</span><br><span style="color: hsl(120, 100%, 40%);">+res_str = ue.verify_metric(max_rate * 0.9, operation='avg', metric='ul_brate', criterion='gt')</span><br><span style="color: hsl(120, 100%, 40%);">+print(res_str + '\n')</span><br><span>diff --git a/suites/4g/suite.conf b/suites/4g/suite.conf</span><br><span>index 352293a..59e393a 100644</span><br><span>--- a/suites/4g/suite.conf</span><br><span>+++ b/suites/4g/suite.conf</span><br><span>@@ -7,3 +7,6 @@</span><br><span> modem:</span><br><span> - times: 1</span><br><span> type: srsue</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+defaults:</span><br><span style="color: hsl(120, 100%, 40%);">+ timeout: 180s</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/17323">change 17323</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/c/osmo-gsm-tester/+/17323"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: osmo-gsm-tester </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: Ib1da58615cdc4f53ac1a27080e94e5b47760c508 </div>
<div style="display:none"> Gerrit-Change-Number: 17323 </div>
<div style="display:none"> Gerrit-PatchSet: 2 </div>
<div style="display:none"> Gerrit-Owner: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins Builder </div>
<div style="display:none"> Gerrit-Reviewer: laforge <laforge@osmocom.org> </div>
<div style="display:none"> Gerrit-Reviewer: pespin <pespin@sysmocom.de> </div>
<div style="display:none"> Gerrit-MessageType: merged </div>