On Sat, Sep 02, 2017 at 10:03:24AM +0200, Harald Welte wrote: When this happened, the root fs was on 4.1G, with a 'quota' of 10G, i.e. probably ~6G taken up by snapshots. Something must have shrunk since then. I've now changed to refquota=5G (so the root fs can be up to 5G, chose 5 since I saw it hit 4G) and removed the quota (i.e. the snapshots will not cause us to limit disk space on the jail). Maybe someone cleared it and that's why we have more space now? Wasn't me. Looking at /var/chache/cgit it doesn't look like it was cleared. I now see NAME USED AVAIL REFER MOUNTPOINT tank/jails/cgit 8.87G 3.25G 1.75G /usr/jails/cgit i.e. 1.75G taken up by root fs, ~7 more G taken up by snapshots. It seems a max of 5G is way beyond the current 1.75G, but I can't tell what grew the root fs to 4G ... I'm happy to accept any other value. Maybe it was the cache that grew this large, and maybe the recent cgit rendering failures were due to hitting disc space limits? ... no, I just cleared the cgit cache and "my" file still renders empty. (cleared by cd /var/cache mkdir not_cgit mv cgit/* not_cgit/ and then testing whether it still works, and it seems to work. So I'm now doing 'rm -rf not_cgit'. For the record root@cgit:/var/cache # du -hs not_cgit/ 1.5G not_cgit/ root@cgit:/var/cache # df -h Filesystem Size Used Avail Capacity Mounted on tank/jails/cgit 5.0G 1.9G 3.1G 38% / root@cgit:/var/cache # rm -rf not_cgit/ root@cgit:/var/cache # df -h Filesystem Size Used Avail Capacity Mounted on tank/jails/cgit 5.0G 368M 4.6G 7% / Seems like we can shrink refquota considerably, and what grew to 4G remains a mystery.) I'm not entirely sure. Holger? [...] Added an initial https://osmocom.org/projects/osmocom-servers/wiki/Osmocomorg_Web_Servers ~N

Today the cgit jail quota again was exceeded and we saw 504 http errors when accessing http://git.osmocom.org/ The reason was indeed that the cache grew to fill the entire 5GB quota. I checked what kind of files were occupying that much cache: Unfortunately more than 150 files each larger than 10MBytes as a result of cgit caching the .tar.gz snapshots it creates. Unfortunately the cgit cache can only be restricted in number of files, not in terms of total size or "don't cache files larger than X". As a workaround I disabled snapshot generation for now. I presume there was some crawler that generated snapshots for lots of commits. Regards, Harald -- - Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6)

Hi, we had enabled snapshots as some of "our" Osmocom developers wanted the feature. And some people continue to clone the website (as if cloning a git repository couldn't be done easier). On my todays flight I came up with a solution but will only implement it the next days (unless someone else is doing it). * disable caching in cgit * enable caching inside the cgit nginx for the majority of URLs. Luckily with git the rendering of a specific commit will not change... * have crawler specific robots.txt to disable the SEO ones.. what do you think? holger -- Holger Freyther <hfreyther(a)sysmocom.de> http://www.sysmocom.de/ ======================================================================= * sysmocom - systems for mobile communications GmbH * Alt-Moabit 93 * 10559 Berlin, Germany * Sitz / Registered office: Berlin, HRB 134158 B * Geschaeftsfuehrer / Managing Directors: Harald Welte