<p>Max has uploaded this change for <strong>review</strong>.</p><p><a href="https://gerrit.osmocom.org/12139">View Change</a></p><pre style="font-family: monospace,monospace; white-space: pre-wrap;">ctrl2cgi: properly limit number of requests<br><br>Manual acquire()/release() of semaphore does not limit number of<br>concurrent requests when combined with explicit yield. Fix this by using<br>semaphore.run() and removing inilineCallbacks decorator.<br><br>Change-Id: I47b8b9f5b726ca0905bb7c023d63b325c7f7d85f<br>Related: SYS#4399<br>---<br>M scripts/ctrl2cgi.py<br>1 file changed, 9 insertions(+), 8 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">git pull ssh://gerrit.osmocom.org:29418/python/osmo-python-tests refs/changes/39/12139/1</pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/scripts/ctrl2cgi.py b/scripts/ctrl2cgi.py</span><br><span>index 1d90ee0..676fc11 100755</span><br><span>--- a/scripts/ctrl2cgi.py</span><br><span>+++ b/scripts/ctrl2cgi.py</span><br><span>@@ -60,6 +60,12 @@</span><br><span> #print('HASH: \nparams="%r"\ninput="%s" \nres="%s"' %(params, input, res))</span><br><span> return res</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+def make_async_req(dst, par, f_write, f_log):</span><br><span style="color: hsl(120, 100%, 40%);">+ d = post(dst, par)</span><br><span style="color: hsl(120, 100%, 40%);">+ d.addCallback(collect, partial(handle_reply, f_write, f_log)) # treq's collect helper is handy to get all reply content at once</span><br><span style="color: hsl(120, 100%, 40%);">+ d.addErrback(lambda e: f_log.critical("HTTP POST error %s while trying to register BSC %s on %s" % (e, par['bsc_id'], dst))) # handle HTTP errors</span><br><span style="color: hsl(120, 100%, 40%);">+ return d</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> class Trap(CTRL):</span><br><span> """</span><br><span> TRAP handler (agnostic to factory's client object)</span><br><span>@@ -93,7 +99,6 @@</span><br><span> self.factory.log.info("Connected to CTRL@%s:%d" % (self.factory.addr_ctrl, self.factory.port_ctrl))</span><br><span> super(CTRL, self).connectionMade()</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- @defer.inlineCallbacks</span><br><span> def handle_locationstate(self, net, bsc, bts, trx, data):</span><br><span> """</span><br><span> Handle location-state TRAP: parse trap content, build CGI Request and use treq's routines to post it while setting up async handlers</span><br><span>@@ -101,13 +106,8 @@</span><br><span> params = make_params(bsc, data)</span><br><span> self.factory.log.debug('location-state@%s.%s.%s.%s (%s) => %s' % (net, bsc, bts, trx, params['time_stamp'], data))</span><br><span> params['h'] = gen_hash(params, self.factory.secret_key)</span><br><span style="color: hsl(0, 100%, 40%);">- d = post(self.factory.location, params)</span><br><span style="color: hsl(0, 100%, 40%);">- d.addCallback(collect, partial(handle_reply, self.transport.write, self.factory.log)) # treq's collect helper is handy to get all reply content at once</span><br><span style="color: hsl(0, 100%, 40%);">- d.addErrback(lambda e, bsc: self.factory.log.critical("HTTP POST error %s while trying to register BSC %s on %s" % (e, bsc, self.factory.location)), bsc) # handle HTTP errors</span><br><span> # Ensure that we run only limited number of requests in parallel:</span><br><span style="color: hsl(0, 100%, 40%);">- yield self.factory.semaphore.acquire()</span><br><span style="color: hsl(0, 100%, 40%);">- yield d # we end up here only if semaphore is available which means it's ok to fire the request without exceeding the limit</span><br><span style="color: hsl(0, 100%, 40%);">- self.factory.semaphore.release()</span><br><span style="color: hsl(120, 100%, 40%);">+ self.factory.semaphore.run(make_async_req, self.factory.location, params, self.transport.write, self.factory.log)</span><br><span> </span><br><span> def handle_notificationrejectionv1(self, net, bsc, bts, trx, data):</span><br><span> """</span><br><span>@@ -121,7 +121,6 @@</span><br><span> Store CGI information so TRAP handler can use it for requests</span><br><span> """</span><br><span> def __init__(self, proto, log):</span><br><span style="color: hsl(0, 100%, 40%);">- self.semaphore = defer.DeferredSemaphore(self.num_max_conn)</span><br><span> self.log = log</span><br><span> level = self.log.getEffectiveLevel()</span><br><span> self.log.setLevel(logging.WARNING) # we do not need excessive debug from lower levels</span><br><span>@@ -155,6 +154,8 @@</span><br><span> T.num_max_conn = config['main'].getint('num_max_conn', T.num_max_conn)</span><br><span> T.secret_key = config['main'].get('secret_key', T.secret_key)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ T.semaphore = defer.DeferredSemaphore(T.num_max_conn)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> log.info("CGI proxy v%s starting with PID %d:" % (__version__, os.getpid()))</span><br><span> log.info("destination %s (concurrency %d)" % (T.location, T.num_max_conn))</span><br><span> log.info("connecting to %s:%d..." % (T.addr_ctrl, T.port_ctrl))</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.osmocom.org/12139">change 12139</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.osmocom.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.osmocom.org/12139"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: python/osmo-python-tests </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-MessageType: newchange </div>
<div style="display:none"> Gerrit-Change-Id: I47b8b9f5b726ca0905bb7c023d63b325c7f7d85f </div>
<div style="display:none"> Gerrit-Change-Number: 12139 </div>
<div style="display:none"> Gerrit-PatchSet: 1 </div>
<div style="display:none"> Gerrit-Owner: Max <msuraev@sysmocom.de> </div>