Jun 262021
 

When upgrading packages on a FreshPorts ingress node (that is usually a jail which processes incoming commits and loads them into the database), I believe there is a possibility of interrupting commit processing. It’s a race condition between pkg removing a file and that file being required by the commit processing.

This post serves to document one way of avoiding this.

The prcocesses

These are expected processes running in the testgit ingress node:

[dan@testgit-ingress01:~] $ ps auwwx
USER         PID %CPU %MEM   VSZ   RSS TT  STAT STARTED    TIME COMMAND
dan        61389  0.2  0.0 13248  4392  1  SJ   17:12   0:00.01 bash
root       61379  0.1  0.0 20632  9600  -  SsJ  17:12   0:00.02 sshd: dan [priv] (sshd)
dan        61382  0.1  0.0 20636  9616  -  SJ   17:12   0:00.01 sshd: dan@pts/1 (sshd)
dan        61383  0.1  0.0 12200  3340  1  SsJ  17:12   0:00.01 -sh (sh)
nagios      8455  0.0  0.0 15960  6392  -  IsJ  11Jun21 0:21.49 /usr/local/sbin/nrpe3 -c /usr/local/etc/nrpe.cfg -d
postfix    22888  0.0  0.0 51464 10744  -  IJ   16:27   0:00.02 pickup -l -t unix -u
freshports 23285  0.0  0.0 11004  2424  -  IsJ  18:52   0:00.04 daemon: freshports[23286] (daemon)
freshports 23286  0.0  0.0 11656  3048  -  SJ   18:52   0:10.78 /bin/sh /usr/local/libexec/freshports-service/freshports.sh
root       42995  0.0  0.0 11376  2772  -  SsJ   2Jun21 0:41.85 /usr/sbin/syslogd -s
root       43331  0.0  0.0 51516  8784  -  IsJ   2Jun21 0:34.10 /usr/local/libexec/postfix/master -w
postfix    43333  0.0  0.0 51568  8836  -  IJ    2Jun21 0:04.54 qmgr -l -t unix -u
ingress    43356  0.0  0.0 11004  2424  -  IsJ   2Jun21 0:01.97 daemon: ingress[43357] (daemon)
ingress    43357  0.0  0.0 11576  2752  -  SJ    2Jun21 1:44.53 /bin/sh /usr/local/libexec/freshports-service/ingress.sh
root       43390  0.0  0.0 19916  8792  -  SsJ   2Jun21 0:07.48 /usr/sbin/sshd
root       43402  0.0  0.0 11308  2632  -  IsJ   2Jun21 0:16.39 /usr/sbin/cron -s
postfix    44636  0.0  0.0 52156 11140  -  IJ    2Jun21 0:02.75 tlsmgr -l -t unix -u
postfix    53480  0.0  0.0 51588 10760  -  IJ   17:09   0:00.01 anvil -l -t unix -u
ingress    61381  0.0  0.0 10676  2188  -  SCJ  17:12   0:00.00 sleep 3
freshports 61430  0.0  0.0 10676  2188  -  SCJ  17:12   0:00.00 sleep 3
dan        61431  0.0  0.0 11704  3064  1  R+J  17:12   0:00.00 ps auwwx
[dan@testgit-ingress01:~] $ 

Of note, nrpe is running in the jail. I suspect I can run that only on the host and inspect the jail.

Some may claim postfix is overkill for running on here, but I have yet to run options which satisfy my [unstated, yes] requirements. But it’s a wonderful MTA and I know how to use it.

There, now I have two upcoming projects:

  1. replace postfix with something smaller on ingress and web nodes
  2. move nrpe from jails to hosts

Stop commit processing

To stop commit the discovery of new commits:

$ sudo service ingress stop
Stopping ingress.

This is a shell script (/usr/local/libexec/freshports-service/ingress.sh): it was carrying out any effective work, it would be interrupted. I don’t know how to properly shutdown this script. In general, it gets a new job created every three minutes, on the 3’s, via a periodic script (see /usr/local/etc/periodic/everythreeminutes/215.fp_check_git_for_commits). If you’re not near a multiple of 3 minutes, and all you see running is the above, I think it’s safe to stop ingress.

To stop the processing of discovered commits (i.e. the output from the ingress daemon):

$ sudo service freshports stop
Stopping freshports.

Now you should see:

 ps auwwx
USER      PID %CPU %MEM   VSZ   RSS TT  STAT STARTED    TIME COMMAND
nagios   8455  0.0  0.0 15960  6392  -  IsJ  11Jun21 0:21.50 /usr/local/sbin/nrpe3 -c /usr/local/etc/nrpe.cfg -d
postfix 22888  0.0  0.0 51464 10744  -  IJ   16:27   0:00.02 pickup -l -t unix -u
root    42995  0.0  0.0 11376  2772  -  SsJ   2Jun21 0:41.87 /usr/sbin/syslogd -s
root    43331  0.0  0.0 51516  8784  -  IsJ   2Jun21 0:34.12 /usr/local/libexec/postfix/master -w
postfix 43333  0.0  0.0 51568  8836  -  IJ    2Jun21 0:04.54 qmgr -l -t unix -u
root    43390  0.0  0.0 19916  8792  -  IsJ   2Jun21 0:07.48 /usr/sbin/sshd
root    43402  0.0  0.0 11308  2632  -  IsJ   2Jun21 0:16.40 /usr/sbin/cron -s
postfix 44636  0.0  0.0 52156 11140  -  IJ    2Jun21 0:02.75 tlsmgr -l -t unix -u
root    61379  0.0  0.0 20632  9600  -  IsJ  17:12   0:00.02 sshd: dan [priv] (sshd)
dan     61382  0.0  0.0 20636  9624  -  SJ   17:12   0:00.05 sshd: dan@pts/1 (sshd)
postfix 84985  0.0  0.0 51588 10760  -  IJ   17:24   0:00.01 anvil -l -t unix -u
dan     61383  0.0  0.0 12200  3340  1  IsJ  17:12   0:00.01 -sh (sh)
dan     61389  0.0  0.0 13256  4396  1  SJ   17:12   0:00.04 bash
dan     92797  0.0  0.0 11696  3060  1  R+J  17:27   0:00.00 ps auwwx

Periodic scripts

All periodic scripts should should observe the OFFLINE variable defined in /usr/local/etc/freshports/config.sh. This variable is set depending on whether or not /usr/local/etc/freshports/OFFLINE exists.

So to effectively block those scripts, issue this command:

$ sudo touch /usr/local/etc/freshports/OFFLINE

After the upgrade

After the upgrade, issue these commands:

$ sudo rm /usr/local/etc/freshports/OFFLINE
$ sudo service freshports start
$ sudo service ingress start

Not perfect

I am sure this is not perfect, but is better than not doing it. It is also best not to do maintenance during these windows:

  1. the top of the hour, when the hourly periodic scripts run.
  2. Just after 0301 UTC when the daily periodic scripts run.
  3. Just after 0415 UTC Saturdays when the weekly periodic scripts run.
  4. Just after 0530 UTC on the first of the month when the monthly periodic scripts run.

I tend to run updates between 1200 and 2000 UTC for personal convenience.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive