This morning I was created with 6 emails, all announcing FAILED: MASTER_PORT, one for each FreshPorts node. Each was sent at 5AM (1000 UTC).
Something happened which upset every one of them. I have no idea why. However, I’m writing this down for future reference.
This announcement came from one of the many system checks run by FreshPorts. In the past, I broke the code for determining the master port of a given port. I then added safety mechanisms to ensure that the next time I break it, I find out quickly. These tests run every hour on the hour.
There are two master port checks (the link goes to the periodic script for that check):
The emails (subject: FAILED: MASTER_PORT) contained:
make -V MASTER_PORT on sysutils/bacula15-client does not give 'sysutils/bacula15-server'\nInstead, it gives ' make: stopped in /'.
I found these log entries on each host:
cd: /usr/ports/sysutils/bacula15-client: No such file or directory make: cannot open /usr/ports/sysutils/bacula15-client/Makefile.
This message is from the checks via make -V script linked above. The following is the script which is invoked by that periodic:
[12:49 dvl-ingress01 dvl ~/scripts] % cat test-master-port.sh
#!/bin/sh
# This scripts invokes the chroot for freshports and works off the Makefile
# see also test-master-port.pl which queries the database.
if [ ! -f config.sh ]
then
echo "config.sh not found by $0"
exit 1
fi
. config.sh
if [ $OFFLINE = 1 ]
then
exit 0
fi
MASTER='sysutils/bacula15-server'
# this always tests against head
COMMAND="/usr/local/bin/sudo /usr/sbin/jexec ${FRESHPORTS_JAIL_NAME} ${FRESHPORTS_JAIL_MASTER_PORT_SCRIPT} ${PORTSDIR} sysutils/bacula15-client"
MASTER_PORT=$(${COMMAND})
if [ "${MASTER_PORT}X" != "${MASTER}X" ]
then
echo "make -V MASTER_PORT on sysutils/bacula15-client does not give '${MASTER}'\nInstead, it gives '${MASTER_PORT}'." | mail -s "FAILED: MASTER_PORT" ${ADMINEMAIL}
fi
More logs
One FreshPorts node (dvl) also had these messages, but I suspect they are an entirely other issue. No other nodes had this particular issue.
Feb 19 09:57:41 dvl-ingress01 FreshPorts[13518]: sleeping for 120 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 09:59:43 dvl-ingress01 FreshPorts[13518]: sleeping for 240 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 10:01:44 dvl-ingress01 FreshPorts[13518]: sleeping for 360 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 10:03:46 dvl-ingress01 FreshPorts[13518]: sleeping for 480 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 10:05:47 dvl-ingress01 FreshPorts[13518]: sleeping for 600 seconds after git checkout failed for (/jails/freshports/usr/ports ... Feb 19 11:55:05 dvl-ingress01 FreshPorts[13518]: sleeping for 7080 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 11:57:07 dvl-ingress01 FreshPorts[13518]: sleeping for 7200 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 11:59:10 dvl-ingress01 FreshPorts[13518]: sleeping for 7320 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 12:01:11 dvl-ingress01 FreshPorts[13518]: sleeping for 7440 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports)
This is the update to multimedia/obs-studio on 2024Q1. Remember that branch name for the next section of his post.
I am not sure why this one node was having issues pulling that commit out of git.
Theory
My theory: it’s a race condition.
Here’s an idea: the script assumes it is running off HEAD, what if it had 2024Q1 checked out (for the above mentioned commit)? sysutils/bacula15-server did not enter the tree until 28 Mar 2024, after 2024Q1 was branched.
The solution: don’t run the checks directly from periodic, instead, queue them up to run after a commit has been process.
Here is a related issue.
Still processing
This was added to the original post at 1306 UTC. That dvl node is still having processing issues. I’ll have to look at it later:
Feb 19 12:03:12 dvl-ingress01 FreshPorts[13518]: sleeping for 7560 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 12:05:14 dvl-ingress01 FreshPorts[13518]: sleeping for 7680 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 12:07:15 dvl-ingress01 FreshPorts[13518]: sleeping for 7800 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) ... Feb 19 12:59:54 dvl-ingress01 FreshPorts[13518]: sleeping for 10920 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports) Feb 19 13:01:55 dvl-ingress01 FreshPorts[13518]: sleeping for 11040 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) (/usr/local/libexec/freshports)
Found it
So I’m not sure how it happened, but it happened. Here’s how I tracked it down. Some output has been modified to avoid scrolling.
[16:53 dvl-ingress01 dvl ~] % ps auwwx | grep perl freshports 13518 0.0 0.0 57920 43080 - IJ 09:57 0:01.38 /usr/local/bin/perl /usr/local/libexec/freshports/load_xml_into_db.pl /var/db/ingress/message-queues/incoming/2025.02.19.09.54.34.000000.d3fd93e6c3568866e6108d0ceb9816f1f8701327.xml
That’s the commit it is still trying to process. Let’s look at that log:
[16:54 dvl-ingress01 dvl /var/db/freshports/message-queues/spooling] % tail -50 2025.02.19.09.54.34.000000.d3fd93e6c3568866e6108d0ceb9816f1f8701327.log ... num of params = 2 /usr/local/bin/git fetch /usr/local/bin/git checkout d3fd93e6c3568866e6108d0ceb9816f1f8701327 error: Your local changes to the following files would be overwritten by checkout: security/vuxml/vuln/2023.xml Please commit your changes or stash them before you switch branches. Aborting sleeping for 24600 seconds after git checkout failed for (/jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327) fetch failed, sleeping... $FreshPorts::Config::Fetch_Retry_Limit='100000000' $numAttempts='99999796' $FreshPorts::Config::Fetch_Sleep_Time='120' about to git checkout = '/usr/local/bin/sudo /usr/local/libexec/freshports/git-checkout.sh /jails/freshports/usr/ports d3fd93e6c3568866e6108d0ceb9816f1f8701327 2>&1'
Ahh, I made some changes in there and did not revert them. That’s a long time ago.
Let’s fix that:
[16:54 dvl-ingress01 dvl /var/db/freshports/message-queues/spooling] % cd /jails/freshports/usr/ports [16:54 dvl-ingress01 dvl /jails/freshports/usr/ports] % git status HEAD detached at 792ecada9ce3 Changes not staged for commit: (use "git add..." to update what will be committed) (use "git restore ..." to discard changes in working directory) modified: security/vuxml/vuln/2023.xml no changes added to commit (use "git add" and/or "git commit -a")
Let’s look at the change, yeah, that looks like something I might have been mucking about with.
[16:54 dvl-ingress01 dvl /jails/freshports/usr/ports] % git diff security/vuxml/vuln/2023.xml diff --git a/security/vuxml/vuln/2023.xml b/security/vuxml/vuln/2023.xml index 74e0306ae776..a87122f33f99 100644 --- a/security/vuxml/vuln/2023.xml +++ b/security/vuxml/vuln/2023.xml @@ -1,64 +1,3 @@ -- - -electron{26,27} -- multiple vulnerabilities -- -- -electron26 -- 26.6.3 - -electron27 -- 27.2.0 - - -Electron developers report:
--- -This update fixes the following vulnerabilities:
--
-- Security: backported fix for CVE-2023-6508.
-- Security: backported fix for CVE-2023-7024.
-- -CVE-2023-6508 -https://github.com/advisories/GHSA-3pr6-6r34-c98x -CVE-2023-7024 -https://github.com/advisories/GHSA-7c6v-f3h8-2x89 -- -2023-12-21 -2023-12-22 -- -gitea -- Prefix Truncation Attack against ChaCha20-Poly1305 and Encrypt-then-MAC aka Terrapin -- -- -gitea -- 1.21.3 - - -The Gitea team reports:
--- -Update golang.org/x/crypto
-- -https://github.com/go-gitea/gitea/releases/tag/v1.21.3 -- -2023-12-19 -2023-12-21 -gitea -- missing permission checks
Let’s restore:
[16:54 dvl-ingress01 dvl /jails/freshports/usr/ports] % sudo git restore security/vuxml/vuln/2023.xml [16:54 dvl-ingress01 dvl /jails/freshports/usr/ports] % git branch * (HEAD detached at 792ecada9ce3) main
What’s the status?
HEAD detached at 792ecada9ce3 Changes not staged for commit: (use "git add/rm..." to update what will be committed) (use "git restore ..." to discard changes in working directory) deleted: .mailmap modified: CHANGES modified: COPYRIGHT modified: GIDs modified: MOVED modified: Makefile deleted: Mk/Features/debuginfo.mk ... devel/libsavitar/files/patch-src_MeshData.cpp devel/libsavitar/files/patch-src_Namespace.cpp devel/libsavitar/files/patch-src_SceneNode.cpp no changes added to commit (use "git add" and/or "git commit -a")
Let’s get out of here.
[16:57 dvl-ingress01 dvl /jails/freshports/usr/ports] % sudo git checkout main Updating files: 100% (14711/14711), done. Previous HEAD position was d3fd93e6c356 multimedia/obs-studio: Broken on armv7 Switched to branch 'main' Your branch is behind 'origin/main' by 46898 commits, and can be fast-forwarded. (use "git pull" to update your local branch)
That’s a lot of commits.
[16:57 dvl-ingress01 dvl /jails/freshports/usr/ports] % sudo git pull Updating e645e190fe78..bbcfadbf2ef1 Updating files: 100% (63843/63843), done. Fast-forward .mailmap | 2 + CHANGES | 121 +- CONTRIBUTING.md | 10 +- COPYRIGHT | 2 +- GIDs | 60 +- MOVED | 1193 +- Makefile | 5 +- Mk/Features/lto.mk | 2 + Mk/Features/pie.mk | 4 + ... x11/yakuake/Makefile | 11 +- x11/yakuake/distinfo | 6 +- x11/yakuake/pkg-plist | 4 +- x11/yelp/Makefile | 4 +- 61823 files changed, 1695414 insertions(+), 1282199 deletions(-) create mode 100644 Mk/Features/testing.mk create mode 100644 Mk/Uses/cl.mk create mode 100644 Mk/Uses/java.mk create mode 100644 Mk/Uses/ocaml.mk ... create mode 100644 x11/xwayland-satellite/Makefile.crates create mode 100644 x11/xwayland-satellite/distinfo create mode 100644 x11/xwayland-satellite/pkg-descr [16:58 dvl-ingress01 dvl /jails/freshports/usr/ports] %
Current status:
[16:58 dvl-ingress01 dvl /jails/freshports/usr/ports] % git status HEAD detached at 55521c087a80 nothing to commit, working tree clean
That status is OK. There is a backlog of 88 commits to get caught up on.
NOTE: this was not production. This was my private working copy.
