I noticed that gerrit places constant load on the server. When I first checked, it was between 30 and 60% CPU load, now is more like 10-15%, but still it is constant load.
I'm pretty sure we're not all clicking around in gerrit all the time, so what would it be:
Gerrit does things in the background, like checking whether unmerged patches have merge conflicts to the current master. I'm not perfectly sure how often this happens, but my guess is that it happens a lot.
We have around 150 open patches on gerrit. If not for sanity, maybe the CPU load on the gerrit server could be another reason to reduce that number.
Related: I notice that the sync of gerrit's git repositories takes longer than it used to. It would be a minute max, while just now I got >10min, and I remember Pau noting something similar recently.
I think we should plain abandon patches that have been idling around for more than 3 months. That would discard about 30, so not that many.
Then we have quite a number that are currently in Merge Conflict...
Seems that we don't get around looking at each one and deciding to kick it or keep it.
~N
Hi Neels,
there are projects with much larger gerrit pending patch queue, so I'd be surprised if we cannot drive it with more patches than we currently do.
On 19. Dec 2017, at 16:56, Harald Welte laforge@gnumonks.org wrote:
Hi Neels,
there are projects with much larger gerrit pending patch queue, so I'd be surprised if we cannot drive it with more patches than we currently do.
I share this sentiment. CPU and RAM are so cheap we should be able to tolerate the ruby and java monster on that system. Let's not abandon patches for "technical limitations".
For the sync. Maybe we need to run git gc in the other repositories as well? Do you have one repo where it takes a lot of time?
holger
On 19/12/17 16:54, Neels Hofmeyr wrote:
Related: I notice that the sync of gerrit's git repositories takes longer than it used to. It would be a minute max, while just now I got >10min, and I remember Pau noting something similar recently.
Correct. For osmo-gsm-tester repo I noticed delays of at least 1 hour to sync pushed branches in gerrit origin to git.osmocom. Same goes for patches merged in gerrit ending up in master in git.osmocom. and I didn't check with detail, but I'd say it could actually take several hours. This didn't happen (or I didn't notice it, but I think I'd have noticed it) prior to 1 or 2 weeks ago.
Hi Pau,
On Wed, Dec 20, 2017 at 12:42:22AM +0100, Pau Espin Pedrol wrote:
Correct. For osmo-gsm-tester repo I noticed delays of at least 1 hour to sync pushed branches in gerrit origin to git.osmocom. Same goes for patches merged in gerrit ending up in master in git.osmocom. and I didn't check with detail, but I'd say it could actually take several hours. This didn't happen (or I didn't notice it, but I think I'd have noticed it) prior to 1 or 2 weeks ago.
So who wants to plot graphs about the latency and feed that into MRTG/grafane or whatever kind of monitoring?
This brings me to a more serious question: Do we have people in this community who would be interested in volunteering as sysadmins for the Osmocom.org infrastructure? There's quite a number of services that we run by now, starting from redmine/gerrit/cgit/jenkins/mailman to the build slaves, etc.
There's also quite a bit of "DevOps" related work pending, such as having Dockerfiles or ansible playbooks to set-up new build slaves, collection of logs/pcaps from the TTCN-3 integration test runs, down to simply having some monitoring in place to monitor the status of the various services and servers [buildslave running out of disk, etc.].
I think this is an idea opportunity for people who are not developers but still want to contribute - or even for developers but those who (think they) don't have the low-level C or telecom skills to work on Osmocom.
Thanks in advance to any interested in contributing in this area.
Regards, Harald
Hi,
This brings me to a more serious question: Do we have people in this community who would be interested in volunteering as sysadmins for the Osmocom.org infrastructure? There's quite a number of services that we run by now, starting from redmine/gerrit/cgit/jenkins/mailman to the build slaves, etc.
Here, I'd like to contribute in these areas. Especially in:
... such as having Dockerfiles or ansible playbooks to set-up new build slaves, collection of logs/pcaps from the TTCN-3 integration test runs, down to simply having some monitoring in place to monitor the status of the various services and servers [buildslave running out of disk, etc.].
But since you mentioned many areas/topics I'd appreciate to discuss the first steps/priorities. We may find some time at the 34c3? I already planned to drop by at the GSM room.
Best Regards, André
On 20/12/2017 15:24, Harald Welte wrote:
Do we have people in this community who would be interested in volunteering as sysadmins for the Osmocom.org infrastructure?
I think this is an idea opportunity for people who are not developers but still want to contribute - or even for developers but those who (think they) don't have the low-level C or telecom skills to work on Osmocom.
Well, with 25+ years experience of sysadmin and some month(s) of low-level C, I can't but raise a hand here! I'm familiar with some of the mentioned infrastructure, not all, same goes probably for the so-called DevOps requirements. but then.. if there's one thing that hasn't changed in 25 years, it's the basic HOWTO process of getting to grips with new tech.
:-)
k/
-
André Boddenberg -
Harald Welte -
Holger Freyther -
Keith -
Neels Hofmeyr -
Pau Espin Pedrol