This post delves into using /usr/ports/INDEX from the FreeBSD port tree for use in solving the http://Mk/bsd.default-versions.mk problem and related issues.
This work was saved on pro05.
I have a script which creates a tab-delimited file from INDEX. The fields are:
origin PKGVERSION PKGNAME
For example:
accessibility/accerciser accerciser-3.48.0 accerciser
Some notes
- INDEX carries one line per flavor, so a flavored port appears several times under one origin.
- Some packages have OSVERSION baked into the version:
open-vm-kmod-13.1.0.1501502,2 vs open-vm-kmod-13.1.0,2 arduino-1.0.6.1501502_8,1 vs arduino-1.0.6_8,1
Those will differ forever unless the INDEX is generated in the same jail FreshPorts refreshes in.
Creating the file
[18:25 dev-ingress01 dvl /usr/local/libexec/freshports] % time python index_pkgversions.py > ~/tmp/INDEX-15.tsv index_pkgversions.py: 38835 rows, 35192 origins, 0 malformed lines python index_pkgversions.py > ~/tmp/INDEX-15.tsv 0.32s user 0.03s system 99% cpu 0.352 total [18:25 dev-ingress01 dvl /usr/local/libexec/freshports] % [18:25 dev-ingress01 dvl /usr/local/libexec/freshports] % wc -l ~/tmp/INDEX-15.tsv 38835 /usr/home/dvl/tmp/INDEX-15.tsv [18:27 dev-ingress01 dvl /usr/local/libexec/freshports] % scp ~/tmp/INDEX-15.tsv pg01: INDEX-15.tsv 100% 2301KB 287.2MB/s 00:00
Over on the database server:
[18:29 pg01 dvl ~] % psql freshports.dev
psql (18.6)
Type "help" for help.
freshports.dev=# CREATE TEMP TABLE index_ports (
origin text NOT NULL,
pkgname text NOT NULL,
package_name text NOT NULL
);
CREATE TABLE
freshports.dev=# \copy index_ports FROM 'index-15.tsv';
COPY 35174
freshports.dev=# CREATE INDEX ON index_ports (origin, pkgname);
CREATE INDEX
freshports.dev=# ANALYZE index_ports;
ANALYZE
freshports.dev=#
Creating the results:
freshports.dev=# \o results.txt
freshports.dev=# WITH fp AS (
SELECT pa.category || '/' || pa.name AS origin,
pa.package_name || '-' || pa.version
|| CASE WHEN COALESCE(pa.revision, '') NOT IN ('', '0')
THEN '_' || pa.revision ELSE '' END
|| CASE WHEN COALESCE(pa.portepoch, '') NOT IN ('', '0')
THEN ',' || pa.portepoch ELSE '' END AS pkgname
FROM ports_active pa
WHERE element_pathname(pa.element_id) LIKE '/ports/head/%'
),
idx AS (
SELECT origin,
count(*) AS packages,
string_agg(pkgname, ' ' ORDER BY pkgname) AS pkgnames
FROM index_ports
GROUP BY origin
)
SELECT COALESCE(fp.origin, idx.origin) AS origin,
CASE WHEN fp.origin IS NULL THEN 'not in FreshPorts'
WHEN idx.origin IS NULL THEN 'not in INDEX'
ELSE 'no matching package'
END AS difference,
fp.pkgname AS freshports_pkgname,
idx.packages AS index_packages,
idx.pkgnames AS index_pkgnames
FROM fp
FULL OUTER JOIN idx ON idx.origin = fp.origin
WHERE fp.origin IS NULL
OR idx.origin IS NULL
OR NOT EXISTS (SELECT 1 FROM index_ports i
WHERE i.origin = fp.origin
AND i.pkgname = fp.pkgname)
ORDER BY difference, origin;
freshports.dev=#
\q
[19:13 pg01 dvl ~] % wc -l results.txt
4847 results.txt
Continuing on from that initial work
It has been several days since the above was written. The script and approach has evolved considerably. These are the steps for find the diverging ports:
determine the latest port commit, call it the Target Commit (TC)
on the ports freshports do
git fetch
git checkout TC
done
run: ./compare-index.sh -b -c TC (this will also do a git fetch and git checkout)
Here’s my sample run from 2026-09-27:
First, get the trees in sync:
[12:19 stage-ingress01 dvl ~] % sudo service freshports stop Stopping freshports. Waiting for PIDS: 55051. [12:20 dev-ingress01 dvl ~] % cd /jails/index/usr/ports [13:10 dev-ingress01 dvl /jails/index/usr/ports] % sudo git checkout 44da815 error: pathspec '44da815' did not match any file(s) known to git [13:10 dev-ingress01 dvl /jails/index/usr/ports] % sudo git fetch remote: Enumerating objects: 33, done. remote: Counting objects: 100% (33/33), done. remote: Compressing objects: 100% (21/21), done. remote: Total 21 (delta 12), reused 0 (delta 0), pack-reused 0 (from 0) Unpacking objects: 100% (21/21), 4.24 KiB | 309.00 KiB/s, done. From https://git.FreeBSD.org/ports 531bc6b3adc1..44da815c7dcc main -> origin/main [13:11 dev-ingress01 dvl /jails/index/usr/ports] % sudo git checkout 44da815 Updating files: 100% (31170/31170), done. Previous HEAD position was dff2312cf5f8 mail/rmilter: Mark deprecated HEAD is now at 44da815c7dcc multimedia/minisatip: Update to 2.0.116 [13:11 dev-ingress01 dvl /jails/index/usr/ports] % cd - /jails/freshports/usr/ports [13:11 dev-ingress01 dvl /jails/freshports/usr/ports] % sudo git checkout 44da815 HEAD is now at 44da815c7dcc multimedia/minisatip: Update to 2.0.116 [13:11 dev-ingress01 dvl /jails/freshports/usr/ports] % sudo git fetch [13:11 dev-ingress01 dvl /jails/freshports/usr/ports] % sudo git checkout 44da815 HEAD is now at 44da815c7dcc multimedia/minisatip: Update to 2.0.116 [13:11 dev-ingress01 dvl /jails/freshports/usr/ports] % sudo service freshports start Starting freshports. [13:50 dev-ingress01 dvl /jails/freshports/usr/ports] %
Then, as the freshports user:
$ ./compare-index.sh -b ./compare-index.sh: building the INDEX from /jails/index/usr/ports as it stands, no commit given ./compare-index.sh: building the INDEX in jail index Generating INDEX-15 - please wait..--- describe.accessibility --- --- describe.arabic --- --- describe.archivers --- --- describe.astro --- --- describe.audio --- --- describe.benchmarks --- --- describe.biology --- --- describe.cad --- --- describe.chinese --- --- describe.comms --- --- describe.converters --- --- describe.databases --- --- describe.deskutils --- --- describe.devel --- --- describe.dns --- --- describe.editors --- --- describe.emulators --- --- describe.finance --- --- describe.french --- --- describe.filesystems --- --- describe.ftp --- --- describe.games --- --- describe.german --- --- describe.graphics --- --- describe.hebrew --- --- describe.hungarian --- --- describe.irc --- --- describe.japanese --- --- describe.java --- --- describe.korean --- --- describe.lang --- --- describe.mail --- --- describe.math --- --- describe.misc --- --- describe.multimedia --- --- describe.net --- --- describe.net-im --- --- describe.net-mgmt --- --- describe.net-p2p --- --- describe.news --- --- describe.polish --- --- describe.ports-mgmt --- --- describe.portuguese --- --- describe.print --- --- describe.russian --- --- describe.science --- --- describe.security --- --- describe.shells --- --- describe.sysutils --- --- describe.textproc --- --- describe.ukrainian --- --- describe.vietnamese --- --- describe.www --- --- describe.x11 --- --- describe.x11-clocks --- --- describe.x11-drivers --- --- describe.x11-fm --- --- describe.x11-fonts --- --- describe.x11-servers --- --- describe.x11-themes --- --- describe.x11-toolkits --- --- describe.x11-wm --- Done../compare-index.sh: INDEX /jails/index/usr/ports/INDEX-15 md5 fd1ebf3802c177b36e14a7262bdeeda3, previously b3625a7dc7924e55bc9df2206e6acd5c ./compare-index.sh: INDEX changed, processing index_pkgversions.py: 38881 rows, 35234 origins, 0 malformed lines ./compare-index.sh: 38881 packages read from the INDEX ./compare-index.sh: 38881 rows loaded and compared 0 /var/db/freshports/cache/spooling/refresh.txt 0 /var/db/freshports/cache/spooling/not-in-index.txt 0 /var/db/freshports/cache/spooling/not-in-freshports.txt $
