Index comparison – deployment notes

These are the deployment notes for the INDEX comparison feature. This paragraph will be updated with the deployment dates as it moves through test, stage, and production. The following was composed by Claude.

FreshPorts now checks its own port versions against the ports INDEX every night, and refreshes the ports which have drifted. Getting there turned up three bugs in how versions and dependencies were recorded, and all of them are fixed in this deploy. Tested on dev-ingress01, freshports.dev and dev.freshports.org.

• Deploy: scripts, modules and database-schema at r6269; freshports-www origin/main at 59a72236
• Database: three changes, two new columns and one data correction. Run them before the code, and the data correction with commit processing paused. See “Deployment order” below.
• Config: new values in both config.pm and config.sh, three new sudoers entries, a new jail, a cron entry and a Nagios check. The scripts name any missing config value and stop, so a forgotten one fails loudly.
• Tested on: dev-ingress01 and freshports.dev. The nightly job has run unattended and reported no differences at two separate commits.
• Rollback to: r6218, and 29804772 for the website. The new columns can stay; nothing reads them when the old code is back.

EXTRACT_DEPENDS and PATCH_DEPENDS were transposed

r6220, r6222, r6223. Files: scripts/Jail/scripts/make-port.sh, database-schema/updates-2026-09-15-extract-patch-depends-swap.ddl

make-port.sh asked for PATCH_DEPENDS at position 33 and EXTRACT_DEPENDS at 34, while the split() in FreshPorts::Port assigned them the other way round. Every port refreshed since has held the two values crossed over: ports.extract_depends held the patch dependencies, ports.patch_depends held the extract ones, and port_dependencies type 'E' meant patch.

• Corrected in make-port.sh rather than in the split, because port.pm already orders extract before patch in six places.
• The data correction is a straight column swap on ports, restricted to the 7277 rows which actually differ — ports carries two AFTER UPDATE ... FOR EACH ROW cache-clearing triggers, so an unqualified UPDATE would have queued the whole tree for re-rendering.
• port_dependencies could not be flipped blindly: its primary key includes the dependency type, and 5209 pairs already carried the same dependent under both 'E' and 'P'. Those need no work — swapped, {E,P} is still {E,P} — and skipping them is what makes the flip collision-free.

BUILD_RUN_DEPENDS is now recorded and displayed

r6219, r6220, r6221, and b2c1a935. Files: modules/port.pm, scripts/Jail/scripts/make-port.sh, classes/ports.php, classes/port-display.php

• New ports.build_run_depends column, and 'A' as a dependency type in port_dependencies — “build And run”. That column is char(1) with no check constraint, so the new type needed no DDL of its own.
• The port page gains a “Build and run dependencies” section, and the reverse dependencies appear under “This port is required by”. Where a port sets BUILD_RUN_DEPENDS, the separate Build and Runtime sections are suppressed: it covers both, and listing all three says the same thing three times.
• Test dependencies also now show under “This port is required by”. ports.test_depends has been populated since 2022 and port_dependencies has been collecting type 'T' rows all along, but the reverse direction was never listed.

PKGVERSION is recorded, because composing it is not always right

r6260, r6261, r6262, r6263, r6264, and 59a72236. Files: modules/port.pm, scripts/Jail/scripts/make-port.sh, scripts/set-pkgversion.pl, classes/ports.php, classes/port-display.php

FreshPorts stored version, revision and portepoch, and everything which needed a package version composed them. For audio/oss that gives 4.2.b2019_5, while the package is oss-4.2.b2019.1501000_5: the kmod framework splices ${OSVERSION} into PKGVERSION, between the version and the revision, where none of the variables we fetched could show it. 103 ports are affected and no amount of composing could ever have reached their real version.

• New ports.pkgversion column, from make -V PKGVERSION, which is exactly what the INDEX carries after the last hyphen of PKGNAME.
• The port page now says so where it happens: audio/oss shows 4.2.b2019_5 (package 4.2.b2019.1501000_5), with a tooltip explaining that packages for other releases of FreeBSD are named differently. Decided by comparing the stored value with what the page composes, not by pattern matching, so a port stops being flagged the moment it stops embedding an OSVERSION.
• set-pkgversion.pl fills the column in for existing rows — about 90 minutes for the tree, resumable, and it asks make for the one value rather than doing a full refresh.
• Three ports — audio/virtual_oss_bluetooth, _equalizer and _sndio — have PORTVERSION set to ${OSVERSION} itself, so their stored version ages with the jail rather than with the port. They still read 1403000, from when the jail was FreeBSD 14.3, because nothing has committed to them since.

The nightly comparison

r6226 through r6269. Files: scripts/index_pkgversions.py, scripts/compare-index.sh, scripts/compare-index-daily.sh, scripts/refresh-listed-ports.pl, scripts/refresh-from-index.sh, scripts/job-waiting.pl, scripts/check_jail_osversion

A port’s version can move without a commit to its own directory: a master port moves, an included Makefile does, or a default version in Mk changes. Nothing told the ingress to refresh those ports, so they sat at an old version indefinitely. The INDEX knows what every port’s version should be, so comparing the two finds them.

• At 02:04 UTC, compare-index-daily.sh reads the freshports/origin/main tag — which git-delta.sh writes after each batch it finishes, so it is the ingress stating how far it has got — and builds the INDEX at that commit in a new index jail. Both sides then describe the same tree, which is the difference between a useful comparison and a list of ports which merely moved in between.
• Four lists come out: ports to refresh, ports where the INDEX is behind the tree, and ports missing from either side. Which way a version moved is decided by pkg version -t, since the query can only compare for equality.
• A non-empty refresh list raises a flag which job-waiting.pl picks up, running refresh-listed-ports.pl. Each port logs the version it held, the version the Makefile gives, and where that version comes from — the master port, the included file, or the Mk default — which is the part that makes the list worth reading.
• A port whose version moved, where every assignment is inside the port and every value is literal, is logged at err: only a commit to that port’s own directory can have moved it, so a commit was missed.
• The INDEX is checksummed, so an unchanged one is not processed twice. Note that gates on one of the two inputs: the database moves independently, so after commit processing catches up the answer can differ while the INDEX has not. -F compares anyway.
• check_jail_osversion is a Nagios check that the index and freshports jails are on the same version of FreeBSD, read the way bsd.port.mk reads OSVERSION. It matters for those 103 ports: each holds the OSVERSION which was true when it was last refreshed, and after a jail upgrade they keep it until their next commit. The check’s header carries the query which lists them and says to feed it to refresh-listed-ports.pl.

Deployment order

Three steps have an ordering constraint; the rest can go in any order.

1. The two new columns, before any code. updates-2026-09-15-build-run-depends.ddl and updates-2026-09-28-pkgversion.ddl. port.pm names both in its UPDATE, so every refresh fails against a database without them.
2. The data correction, with commit processing paused. Pause, run updates-2026-09-15-extract-patch-depends-swap.ddl, deploy make-port.sh, resume. Deploying first means anything processed in between is written correctly and then transposed by the migration. Running the migration twice swaps back, so it is its own rollback.
3. port.pm and make-port.sh together. The split is positional — 49 -V variables against 49 variables — so a mismatched pair assigns every field to the wrong column, silently.
4. Config, sudoers, the index jail, the cron entry and the Nagios check. Then set-pkgversion.pl for the backfill, which is worth doing with ports_clear_cache disabled: 35,000 updates would otherwise queue 35,000 pages for re-rendering, for a column nothing displays.

NOTE:

1. A new index jail must be deployed.
2. The sudoer permissions need to be updated via this command: ansible-playbook freshports-ingress.yml –tags=sudoers

Smoke tests after deploy

Check	Expect
/audio/oss/	4.2.b2019_5 (package 4.2.b2019.1501000_5)
Any port which does not embed an OSVERSION	The version alone, with no (package …)
A port with BUILD_RUN_DEPENDS	A “Build and run dependencies” section, and no separate Build or Runtime section
A port with only test dependencies	Its Test section, and not “This port has no dependencies.”
check_jail_osversion	OSVERSION OK - index and freshports both …, exit 0. Stop one jail’s param.h being readable to see the UNKNOWN path.
compare-index-daily.sh by hand	Four zero counts, mail to $ADMINEMAIL, watchgoose green. Allow 15–20 minutes for the INDEX build.
SELECT count(*) FROM ports WHERE pkgversion IS NULL	0 for active head ports, once the backfill has run

Rollback

Code back to r6218 and 29804772. Remove the cron entry and the Nagios check. The two new columns can stay — nothing reads them with the old code in place — and the EXTRACT/PATCH data correction should stay too, since the old make-port.sh is what transposed them in the first place. If it does need undoing, run that migration a second time.