p5-freshports-scripts changes for 2.3.0

p5-freshports-scripts is being updated to 2.3.0 – this post outlines the changes. See also p5-freshports-modules changes for 2.3.0

FreshPorts now checks its own port versions against the ports INDEX every night, and refreshes the ports which have drifted. That is almost all of this release. It also starts recording PKGVERSION from make, because composing a package version from what we stored was wrong for about a hundred ports, and adds a Nagios check which guards the assumption the comparison rests on.

Release: ingress/scripts 2.3.0, r6225 through r6269. Previous release 2.2.7 was r6224.
Requires: ingress/modules at r6270 or later, and the ports.pkgversion column from database-schema r6262. Both before this package. See “Installing” below.
New dependency: python3, on the host rather than in a jail.
Config: five new values in config.sh, three new sudoers entries, one new jail, one cron entry, one Nagios check. A missing config value is named and the script exits at the top, so a forgotten one fails loudly.
Tested on: dev-ingress01. The nightly job has run unattended and reported no differences at two separate commits.
Rollback to: 2.2.7. The new column can stay; nothing reads it with the old code in place.

What is in the package

Seven new files:

index_pkgversions.py — parse an INDEX into origin, PKGNAME and package name
compare-index.sh — compare the database against the INDEX
compare-index-daily.sh — run that nightly and mail the result
refresh-from-index.sh — the job job-waiting.pl runs
refresh-listed-ports.pl — refresh the ports named in a file
set-pkgversion.pl — one-off backfill of ports.pkgversion
check_jail_osversion — Nagios check that both jails run the same FreeBSD

Three modified: config.sh.sample, job-waiting.pl (one line), Jail/scripts/make-port.sh.

The nightly comparison

r6226–r6229, r6231, r6232, r6235, r6238, r6240–r6242, r6253, r6254, r6257–r6259, r6267, r6269. Files: index_pkgversions.py, compare-index.sh, compare-index-daily.sh, refresh-from-index.sh, job-waiting.pl

A port’s version can move without a commit to its own directory: a master port moves, an included Makefile does, or a default version in Mk changes. Nothing told the ingress to refresh those ports, so they sat at an old version indefinitely. The INDEX knows what every port’s version should be, so comparing the two finds them.

At 02:04 UTC, compare-index-daily.sh reads the freshports/origin/main tag — which git-delta.sh writes only after a batch has finished, so it is the ingress stating how far it has got — and builds the INDEX at that commit. Both sides then describe the same tree, which is the difference between a useful comparison and a list of ports which merely moved in between.
Four lists come out: ports to refresh, ports where the INDEX is behind the tree, and ports missing from either side. Which way a version moved is decided by pkg version -t, since the query can only compare for equality.
Only the version is compared, never the package name. A port’s PKGNAMEPREFIX follows DEFAULT_VERSIONS, so py311-foo and py312-foo are the same port at the same version, and comparing names reported thousands of ports which needed no work.
A non-empty refresh list raises a flag which job-waiting.pl picks up, running refresh-from-index.sh. Scheduled rather than run in the cron job: a refresh writes to the database and clears caches, which belongs in the queue with everything else of that kind.
The INDEX is checksummed, so an unchanged one is not processed twice. That gates on one of the two inputs only — the database moves independently, so after commit processing catches up the answer can differ while the INDEX has not. -F compares anyway.
Mail goes out whether the run worked or not. A silent failure at 02:04 is one nobody finds until the lists go stale. On success it also pings watchgoose, so a run which stops happening is noticed.

Why a port’s version moved, not just that it did

r6234, r6236, r6237, r6239, r6243–r6249, r6252. File: refresh-listed-ports.pl

Nothing existing would refresh a list of ports. refresh-one-port.pl takes no arguments — the port is hardcoded in its SQL. refresh-each-port.pl does the whole tree and would die before reaching any of it. So this follows process_default_versions.pl, which is current and works.

Each port logs the version it held, the version the Makefile gives, and where that version comes from — the master port, the included file, or the Mk default — which is the part that makes the list worth reading.
Mk‘s own assignments are skipped when scanning. bsd.port.mk derives PORTVERSION from DISTVERSION and back again, and reporting those lines was accurate but useless: they are true of every port in the tree, so they explained nothing about this one and buried the line which did. Mk is still searched when following a reference, because PYTHON_DEFAULT really is set in bsd.default-versions.mk and that is an answer.
A port whose version moved, where every assignment is inside the port and every value is literal, is logged at err: only a commit to that port’s own directory can have moved it, so a commit was missed.
Each port is committed on its own. A list of a couple of hundred is long enough that a failure part way through should not discard the ports already done. A port which fails is rolled back, counted and skipped, and the run ends with a tally.
--dryrun and --debug are plain flags, and a dry run reports the provenance without writing anything.

PKGVERSION, because composing it is not always right

r6261, r6263, r6264, r6265. Files: Jail/scripts/make-port.sh, set-pkgversion.pl, compare-index.sh

FreshPorts stored version, revision and portepoch, and everything which needed a package version composed them. For audio/oss that gives 4.2.b2019_5, while the package is oss-4.2.b2019.1501000_5: the kmod framework splices ${OSVERSION} into PKGVERSION, between the version and the revision, where none of the variables we fetched could show it. 103 ports are affected and no amount of composing could ever have reached their real version.

make-port.sh now asks for PKGVERSION, which is exactly what the INDEX carries after the last hyphen of PKGNAME. The comparison uses that column rather than composing one.
set-pkgversion.pl fills the column in for existing rows — about 90 minutes for the tree, resumable, asking make for the one value rather than doing a full refresh.
Worth running with ports_clear_cache disabled: 35,000 updates would otherwise queue 35,000 pages for re-rendering, for a column nothing displayed until the website side of this work shipped.

check_jail_osversion

r6266, r6268. File: check_jail_osversion

The database side is built in the freshports jail and the INDEX in the index jail. For those 103 ports the two agree only while both jails report the same OSVERSION; let them drift apart and the ports start showing as differences for a reason which has nothing to do with a port having changed. The check reads __FreeBSD_version from each jail’s sys/param.h, the way bsd.port.mk reads it, so it needs no privileges — it only reads a file.

The header carries a “when this goes red” block, because the fix is not obvious: those ports hold the OSVERSION which was true when each was last refreshed, and after a jail upgrade they keep it until their next commit, which for some is years away. compare-index.sh will not catch it — it takes the OSVERSION out of both sides before comparing, which is the component that just changed. The block gives the query which lists them and says to feed it to refresh-listed-ports.pl, not set-pkgversion.pl: a few of these ports have PORTVERSION set to ${OSVERSION} itself, so their version column ages too, and only a refresh puts both right.

Error detection and performance

r6230, r6233, r6250, r6251, r6255, r6256. File: compare-index.sh

A run which failed printed a single line and stopped, with no hint of why: every failure path went to syslog, so at a terminal the script looked like it had simply finished. Every message now reaches stderr or stdout as well.
psql gains ON_ERROR_STOP. Without it a failed statement is reported on stderr and psql still exits 0, so a broken query, a missing table or a refused connection each produced an empty result and a run which looked clean with no differences to report.
The row count loaded is compared with the line count of the file just parsed. That is the check which would have caught comparing against an INDEX we had not just read.
The branch restriction was calling element_pathname(), which walks up the element tree one query per level, once per row — about 35,000 times a run. ports_active already hands back a pathname column.
The comparison itself took 243 seconds: a materialised CTE was being re-scanned per row. It is now 1,676 ms.

Installing

Two steps have an ordering constraint; the rest can go in any order.

ports.pkgversion before any code. database-schema r6262. port.pm names the column in its UPDATE, so every refresh fails against a database without it.
modules and this package together. Jail/scripts/make-port.sh is positionally coupled to port.pm r6260: the make -V output is split by position, 49 variables against 49, so a mismatched pair assigns every field to the wrong column, silently. Deploying either alone corrupts every port it touches.
Install python3. index_pkgversions.py runs on the host.
Create the index jail, with perl installed and its own ports tree synchronised with the freshports one. make index needs perl and the freshports jail has no packages installed, which is why this is a second jail rather than the existing one.
Add the five new config.sh values: SUDO, INDEX_JAIL_NAME, INDEX_JAIL_BASE_DIR, WATCHGOOSE_COMPARE_INDEX, REFRESHFROMINDEXFLAG. The last must be the same path as $FreshPorts::Config::RefreshFromIndexFlag in config.pm.
Add the three sudoers entries. They supersede the two from r6229 and r6231, which named the freshports jail and are no longer used.
Add the cron entry. It names the user, so it goes in a crontab which has a user field — /etc/crontab or a file under cron.d — and it needs CRON_TZ=UTC, since 02:04 was asked for in UTC and cron otherwise uses the machine’s own time.
Install check_jail_osversion as a Nagios plugin, and run set-pkgversion.pl for the backfill.

The sudoers entries, all for the index jail. Unlike the existing freshports entries these take nothing from outside the script except the commit hash, so the argument lists are fixed:

freshports ALL=(ALL) NOPASSWD:/usr/local/bin/git -C /jails/index/usr/ports fetch
freshports ALL=(ALL) NOPASSWD:/usr/local/bin/git -C /jails/index/usr/ports checkout *
freshports ALL=(ALL) NOPASSWD:/usr/sbin/jexec index /usr/bin/make -C /usr/ports index

If this host is coming from a release earlier than 2.2.7, that release’s EXTRACT_DEPENDS/PATCH_DEPENDS data migration and ports.build_run_depends column are prerequisites too.

Smoke tests after install

Check	Expect
`check_jail_osversion`	`OSVERSION OK - index and freshports both …`, exit 0
`index_pkgversions.py \| head`	Tab separated origin, PKGNAME, package name; a summary of rows, origins and malformed lines on stderr
`compare-index.sh` with no `-b`	Compares against the existing INDEX, reports the four bucket counts, exits 0. Confirms the database connection and that the reading role may create temp objects.
`refresh-listed-ports.pl --dryrun` on a short list	Per-port provenance, no database writes
`compare-index-daily.sh` by hand	Four zero counts, mail to `$ADMINEMAIL`, watchgoose green. Allow 15–20 minutes for the INDEX build.
`SELECT count(*) FROM ports WHERE pkgversion IS NULL`	0 for active head ports, once the backfill has run
A refresh of `audio/oss`	`pkgversion` = `4.2.b2019.1501000_5`, not `4.2.b2019_5`

A missing config value should name itself and stop at the top of the script. Worth confirming once, by commenting one out, rather than discovering it inside jexec with no jail name.

Rollback

Back to 2.2.7, with modules back to r6259 — the two move together in either direction, for the same reason they install together. Remove the cron entry and the Nagios check. ports.pkgversion can stay: nothing reads it with the old code in place, and leaving it means the backfill is not lost if this is reinstalled. The index jail can stay too; nothing else uses it, and nothing else is harmed by it.