When upgrading packages on a FreshPorts ingress node (that is usually a jail which processes incoming commits and loads them into the database), I believe there is a possibility of interrupting commit processing. It’s a race condition between pkg removing a file and that file being required by the commit processing.
This post serves to document one way of avoiding this.
The prcocesses
These are expected processes running in the testgit ingress node:
[dan@testgit-ingress01:~] $ ps auwwx USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND dan 61389 0.2 0.0 13248 4392 1 SJ 17:12 0:00.01 bash root 61379 0.1 0.0 20632 9600 - SsJ 17:12 0:00.02 sshd: dan [priv] (sshd) dan 61382 0.1 0.0 20636 9616 - SJ 17:12 0:00.01 sshd: dan@pts/1 (sshd) dan 61383 0.1 0.0 12200 3340 1 SsJ 17:12 0:00.01 -sh (sh) nagios 8455 0.0 0.0 15960 6392 - IsJ 11Jun21 0:21.49 /usr/local/sbin/nrpe3 -c /usr/local/etc/nrpe.cfg -d postfix 22888 0.0 0.0 51464 10744 - IJ 16:27 0:00.02 pickup -l -t unix -u freshports 23285 0.0 0.0 11004 2424 - IsJ 18:52 0:00.04 daemon: freshports[23286] (daemon) freshports 23286 0.0 0.0 11656 3048 - SJ 18:52 0:10.78 /bin/sh /usr/local/libexec/freshports-service/freshports.sh root 42995 0.0 0.0 11376 2772 - SsJ 2Jun21 0:41.85 /usr/sbin/syslogd -s root 43331 0.0 0.0 51516 8784 - IsJ 2Jun21 0:34.10 /usr/local/libexec/postfix/master -w postfix 43333 0.0 0.0 51568 8836 - IJ 2Jun21 0:04.54 qmgr -l -t unix -u ingress 43356 0.0 0.0 11004 2424 - IsJ 2Jun21 0:01.97 daemon: ingress[43357] (daemon) ingress 43357 0.0 0.0 11576 2752 - SJ 2Jun21 1:44.53 /bin/sh /usr/local/libexec/freshports-service/ingress.sh root 43390 0.0 0.0 19916 8792 - SsJ 2Jun21 0:07.48 /usr/sbin/sshd root 43402 0.0 0.0 11308 2632 - IsJ 2Jun21 0:16.39 /usr/sbin/cron -s postfix 44636 0.0 0.0 52156 11140 - IJ 2Jun21 0:02.75 tlsmgr -l -t unix -u postfix 53480 0.0 0.0 51588 10760 - IJ 17:09 0:00.01 anvil -l -t unix -u ingress 61381 0.0 0.0 10676 2188 - SCJ 17:12 0:00.00 sleep 3 freshports 61430 0.0 0.0 10676 2188 - SCJ 17:12 0:00.00 sleep 3 dan 61431 0.0 0.0 11704 3064 1 R+J 17:12 0:00.00 ps auwwx [dan@testgit-ingress01:~] $
Of note, nrpe is running in the jail. I suspect I can run that only on the host and inspect the jail.
Some may claim postfix is overkill for running on here, but I have yet to run options which satisfy my [unstated, yes] requirements. But it’s a wonderful MTA and I know how to use it.
There, now I have two upcoming projects:
- replace postfix with something smaller on ingress and web nodes
- move nrpe from jails to hosts
Stop commit processing
To stop commit the discovery of new commits:
$ sudo service ingress stop Stopping ingress.
This is a shell script (/usr/local/libexec/freshports-service/ingress.sh): it was carrying out any effective work, it would be interrupted. I don’t know how to properly shutdown this script. In general, it gets a new job created every three minutes, on the 3’s, via a periodic script (see /usr/local/etc/periodic/everythreeminutes/215.fp_check_git_for_commits). If you’re not near a multiple of 3 minutes, and all you see running is the above, I think it’s safe to stop ingress.
To stop the processing of discovered commits (i.e. the output from the ingress daemon):
$ sudo service freshports stop Stopping freshports.
Now you should see:
ps auwwx USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND nagios 8455 0.0 0.0 15960 6392 - IsJ 11Jun21 0:21.50 /usr/local/sbin/nrpe3 -c /usr/local/etc/nrpe.cfg -d postfix 22888 0.0 0.0 51464 10744 - IJ 16:27 0:00.02 pickup -l -t unix -u root 42995 0.0 0.0 11376 2772 - SsJ 2Jun21 0:41.87 /usr/sbin/syslogd -s root 43331 0.0 0.0 51516 8784 - IsJ 2Jun21 0:34.12 /usr/local/libexec/postfix/master -w postfix 43333 0.0 0.0 51568 8836 - IJ 2Jun21 0:04.54 qmgr -l -t unix -u root 43390 0.0 0.0 19916 8792 - IsJ 2Jun21 0:07.48 /usr/sbin/sshd root 43402 0.0 0.0 11308 2632 - IsJ 2Jun21 0:16.40 /usr/sbin/cron -s postfix 44636 0.0 0.0 52156 11140 - IJ 2Jun21 0:02.75 tlsmgr -l -t unix -u root 61379 0.0 0.0 20632 9600 - IsJ 17:12 0:00.02 sshd: dan [priv] (sshd) dan 61382 0.0 0.0 20636 9624 - SJ 17:12 0:00.05 sshd: dan@pts/1 (sshd) postfix 84985 0.0 0.0 51588 10760 - IJ 17:24 0:00.01 anvil -l -t unix -u dan 61383 0.0 0.0 12200 3340 1 IsJ 17:12 0:00.01 -sh (sh) dan 61389 0.0 0.0 13256 4396 1 SJ 17:12 0:00.04 bash dan 92797 0.0 0.0 11696 3060 1 R+J 17:27 0:00.00 ps auwwx
Periodic scripts
All periodic scripts should should observe the OFFLINE variable defined in /usr/local/etc/freshports/config.sh. This variable is set depending on whether or not /usr/local/etc/freshports/OFFLINE exists.
So to effectively block those scripts, issue this command:
$ sudo touch /usr/local/etc/freshports/OFFLINE
After the upgrade
After the upgrade, issue these commands:
$ sudo rm /usr/local/etc/freshports/OFFLINE $ sudo service freshports start $ sudo service ingress start
Not perfect
I am sure this is not perfect, but is better than not doing it. It is also best not to do maintenance during these windows:
- the top of the hour, when the hourly periodic scripts run.
- Just after 0301 UTC when the daily periodic scripts run.
- Just after 0415 UTC Saturdays when the weekly periodic scripts run.
- Just after 0530 UTC on the first of the month when the monthly periodic scripts run.
I tend to run updates between 1200 and 2000 UTC for personal convenience.
