28 Jan
2019
28 Jan
'19
9:08 a.m.
A wild guess but can any of you check if the GSM tester state is corrupt again? My test
with virtual equipment failed the last two times I tried with no IP addresses available
and the production run seems broken as well.
thank you!
holger
28 Jan
28 Jan
2:10 p.m.
On Mon, Jan 28, 2019 at 08:08:36AM +0000, Holger Freyther wrote:
A wild guess but can any of you check if the GSM
tester state is corrupt again? My test with virtual equipment failed the last two times I
tried with no IP addresses available and the production run seems broken as well.
thank you!
I notice the osmo-gsmtester-prod moved to IP address .107 since I was last on it.
But I can log in.
The /var/tmp/osmo-gsm-tester/state has reserved resources since 2019-01-23 22:21:01.
The 'ps' output looks troubling, a large number of tcpdumps running: 843!! wtf
See http://people.osmocom.org/neels/eengeeK2/gsm_tester_ps.txt
No 'osmo-gsm-tester.py' in ps, so nothing is running.
I erased the reserved_resources and rebooted.
~N
9:30 p.m.
Hi,
that happens from time to time because while running a job in there, the
jenkins connection between jenkins master and slave fails. As a result,
jenkins decides it's a good idea to kill -9 osmo-gsm-tester process,
leaving as a result its child processes alive without anybody
controlling them.
I usually take care myself of removing those processes (ps -ef | grep
osmo, kill) and cleaning state (rm /var/tmp/osmo-gsm-tester/state/*)
when I see lots of tests failing with UNKNOWN during the more-or-less
daily jenkins job. Unfortunately, due to being on sick leave these days
it may take longer than usual to have this work done after a conn
failure, my apologies.
Thanks Neels for taking care of the issue.
Kind regards,
Pau
--
- Pau Espin Pedrol <pespin(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
29 Jan
29 Jan
1:33 a.m.
New subject: Jeknins leaking processes (was Re: Reset GSM tester state?)
Hi Pau,
On Mon, Jan 28, 2019 at 09:30:36PM +0100, Pau Espin Pedrol wrote:
that happens from time to time because while running a
job in there, the
jenkins connection between jenkins master and slave fails. As a result,
jenkins decides it's a good idea to kill -9 osmo-gsm-tester process,
leaving as a result its child processes alive without anybody controlling
them.
FYI, this leaking ofprocesses is a known bug in Jenkins since 2013 (!)
which many people raised at
https://issues.jenkins-ci.org/browse/JENKINS-17116
It's a real pity that such an importan bug doesn't seem to get fixed
by Jenkins upstream. If there are any Java developers with some spare
cycles reading here, it would really be great if you could contribue a
related fix upstream. Thanks!
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
5:25 p.m.
On 28. Jan 2019, at 20:30, Pau Espin Pedrol
<pespin(a)sysmocom.de> wrote:
Hi,
Hi!
I don't know how the state works but have you considered:
* Adding stale detection, either in code or jenkins?
* far fetched.. look if subreaper can be of any help?
cheers
holger
that happens from time to time because while running a job in there, the jenkins
connection between jenkins master and slave fails. As a result, jenkins decides it's a
good idea to kill -9 osmo-gsm-tester process,
leaving as a result its child processes alive without anybody controlling them.
I usually take care myself of removing those processes (ps -ef | grep osmo, kill) and
cleaning state (rm /var/tmp/osmo-gsm-tester/state/*) when I see lots of tests failing with
UNKNOWN during the more-or-less daily jenkins job. Unfortunately, due to being on sick
leave these days it may take longer than usual to have this work done after a conn
failure, my apologies.
Thanks Neels for taking care of the issue.
Kind regards,
Pau
--
- Pau Espin Pedrol <pespin(a)sysmocom.de> http://www.sysmocom.de/
=======================================================================
* sysmocom - systems for mobile communications GmbH
* Alt-Moabit 93
* 10559 Berlin, Germany
* Sitz / Registered office: Berlin, HRB 134158 B
* Geschaeftsfuehrer / Managing Director: Harald Welte
30 Jan
30 Jan
1:47 p.m.
On Tue, Jan 29, 2019 at 04:25:08PM +0000, Holger Freyther wrote:
The state works pretty nicely if you don't kill -9 the process :P
The thought to build another safeguard around it has come up a few times
before.
yet we probably wouldn't be talking about it if jenkins didn't maintain that
habit of just kicking open the airlock to void space on the osmo-gsm-tester.
(OTOH of course it would be even nicer if we could deal with kill -9 safely as
well, but where do you draw the line; there's always *some* place where a kill
-9 breaks the plan.)
~N
On 28. Jan 2019, at 20:30, Pau Espin Pedrol
<pespin(a)sysmocom.de> wrote:
Hi,
Hi!
I don't know how the state works but have you considered:
* Adding stale detection, either in code or jenkins?
* far fetched.. look if subreaper can be of any help?
2243
days inactive
2245
days old
5 comments
4 participants
participants (4)
-
Harald Welte -
Holger Freyther -
Neels Hofmeyr -
Pau Espin Pedrol