using INDEX to find PORTVERSION deviations

Leave a Comment / By /

This post delves into using /usr/ports/INDEX from the FreeBSD port tree for use in solving the http://Mk/bsd.default-versions.mk problem and related issues.

This work was saved on pro05.

I have a script which creates a tab-delimited file from INDEX. The fields are:

origin PKGVERSION PKGNAME

For example:

accessibility/accerciser	accerciser-3.48.0	accerciser

Some notes

  1. INDEX carries one line per flavor, so a flavored port appears several times under one origin.
  2. Some packages have OSVERSION baked into the version: 
    open-vm-kmod-13.1.0.1501502,2   vs   open-vm-kmod-13.1.0,2
arduino-1.0.6.1501502_8,1       vs   arduino-1.0.6_8,1

    Those will differ forever unless the INDEX is generated in the same jail FreshPorts refreshes in.

Creating the file

[18:25 dev-ingress01 dvl /usr/local/libexec/freshports] % time python index_pkgversions.py > ~/tmp/INDEX-15.tsv
index_pkgversions.py: 38835 rows, 35192 origins, 0 malformed lines
python index_pkgversions.py > ~/tmp/INDEX-15.tsv  0.32s user 0.03s system 99% cpu 0.352 total
[18:25 dev-ingress01 dvl /usr/local/libexec/freshports] % 
[18:25 dev-ingress01 dvl /usr/local/libexec/freshports] % wc -l ~/tmp/INDEX-15.tsv
   38835 /usr/home/dvl/tmp/INDEX-15.tsv
[18:27 dev-ingress01 dvl /usr/local/libexec/freshports] % scp ~/tmp/INDEX-15.tsv pg01:
INDEX-15.tsv               100% 2301KB 287.2MB/s   00:00

Over on the database server:

[18:29 pg01 dvl ~] % psql freshports.dev
psql (18.6)
Type "help" for help.

freshports.dev=# CREATE TEMP TABLE index_ports (
    origin       text NOT NULL,
    pkgname      text NOT NULL,
    package_name text NOT NULL
);
CREATE TABLE
freshports.dev=# \copy index_ports FROM 'index-15.tsv';
COPY 35174
freshports.dev=# CREATE INDEX ON index_ports (origin, pkgname);
CREATE INDEX
freshports.dev=# ANALYZE index_ports;
ANALYZE
freshports.dev=#

Creating the results:

freshports.dev=# \o results.txt
freshports.dev=# WITH fp AS (
    SELECT pa.category || '/' || pa.name AS origin,
           pa.package_name || '-' || pa.version
             || CASE WHEN COALESCE(pa.revision,  '') NOT IN ('', '0')
                     THEN '_' || pa.revision  ELSE '' END
             || CASE WHEN COALESCE(pa.portepoch, '') NOT IN ('', '0')
                     THEN ',' || pa.portepoch ELSE '' END AS pkgname
      FROM ports_active pa
     WHERE element_pathname(pa.element_id) LIKE '/ports/head/%'
),
idx AS (
    SELECT origin,
           count(*) AS packages,
           string_agg(pkgname, ' ' ORDER BY pkgname) AS pkgnames
      FROM index_ports
     GROUP BY origin
)
SELECT COALESCE(fp.origin, idx.origin) AS origin,
       CASE WHEN fp.origin  IS NULL THEN 'not in FreshPorts'
            WHEN idx.origin IS NULL THEN 'not in INDEX'
            ELSE 'no matching package'
       END AS difference,
       fp.pkgname   AS freshports_pkgname,
       idx.packages AS index_packages,
       idx.pkgnames AS index_pkgnames
  FROM fp
  FULL OUTER JOIN idx ON idx.origin = fp.origin
 WHERE fp.origin  IS NULL
    OR idx.origin IS NULL
    OR NOT EXISTS (SELECT 1 FROM index_ports i
                    WHERE i.origin  = fp.origin
                      AND i.pkgname = fp.pkgname)
 ORDER BY difference, origin;
freshports.dev=# 
\q
[19:13 pg01 dvl ~] % wc -l results.txt 
    4847 results.txt

Continuing on from that initial work

It has been several days since the above was written. The script and approach has evolved considerably. These are the steps for find the diverging ports:

determine the latest port commit, call it the Target Commit (TC)
on the ports freshports do
  git fetch
  git checkout TC
done

run: ./compare-index.sh -b -c TC (this will also do a git fetch and git checkout)

Here’s my sample run from 2026-09-27:

First, get the trees in sync:

[12:19 stage-ingress01 dvl ~] % sudo service freshports stop
Stopping freshports.
Waiting for PIDS: 55051.
[12:20 dev-ingress01 dvl ~] % cd /jails/index/usr/ports
[13:10 dev-ingress01 dvl /jails/index/usr/ports] % sudo git checkout 44da815
error: pathspec '44da815' did not match any file(s) known to git
[13:10 dev-ingress01 dvl /jails/index/usr/ports] % sudo git fetch
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 21 (delta 12), reused 0 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (21/21), 4.24 KiB | 309.00 KiB/s, done.
From https://git.FreeBSD.org/ports
   531bc6b3adc1..44da815c7dcc  main       -> origin/main
[13:11 dev-ingress01 dvl /jails/index/usr/ports] % sudo git checkout 44da815
Updating files: 100% (31170/31170), done.
Previous HEAD position was dff2312cf5f8 mail/rmilter: Mark deprecated
HEAD is now at 44da815c7dcc multimedia/minisatip: Update to 2.0.116
[13:11 dev-ingress01 dvl /jails/index/usr/ports] % cd -
/jails/freshports/usr/ports
[13:11 dev-ingress01 dvl /jails/freshports/usr/ports] % sudo git checkout 44da815
HEAD is now at 44da815c7dcc multimedia/minisatip: Update to 2.0.116
[13:11 dev-ingress01 dvl /jails/freshports/usr/ports] % sudo git fetch           
[13:11 dev-ingress01 dvl /jails/freshports/usr/ports] % sudo git checkout 44da815
HEAD is now at 44da815c7dcc multimedia/minisatip: Update to 2.0.116
[13:11 dev-ingress01 dvl /jails/freshports/usr/ports] % sudo service freshports start
Starting freshports.
[13:50 dev-ingress01 dvl /jails/freshports/usr/ports] %

Then, as the freshports user:

$ ./compare-index.sh -b
./compare-index.sh: building the INDEX from /jails/index/usr/ports as it stands, no commit given
./compare-index.sh: building the INDEX in jail index
Generating INDEX-15 - please wait..--- describe.accessibility ---
--- describe.arabic ---
--- describe.archivers ---
--- describe.astro ---
--- describe.audio ---
--- describe.benchmarks ---
--- describe.biology ---
--- describe.cad ---
--- describe.chinese ---
--- describe.comms ---
--- describe.converters ---
--- describe.databases ---
--- describe.deskutils ---
--- describe.devel ---
--- describe.dns ---
--- describe.editors ---
--- describe.emulators ---
--- describe.finance ---
--- describe.french ---
--- describe.filesystems ---
--- describe.ftp ---
--- describe.games ---
--- describe.german ---
--- describe.graphics ---
--- describe.hebrew ---
--- describe.hungarian ---
--- describe.irc ---
--- describe.japanese ---
--- describe.java ---
--- describe.korean ---
--- describe.lang ---
--- describe.mail ---
--- describe.math ---
--- describe.misc ---
--- describe.multimedia ---
--- describe.net ---
--- describe.net-im ---
--- describe.net-mgmt ---
--- describe.net-p2p ---
--- describe.news ---
--- describe.polish ---
--- describe.ports-mgmt ---
--- describe.portuguese ---
--- describe.print ---
--- describe.russian ---
--- describe.science ---
--- describe.security ---
--- describe.shells ---
--- describe.sysutils ---
--- describe.textproc ---
--- describe.ukrainian ---
--- describe.vietnamese ---
--- describe.www ---
--- describe.x11 ---
--- describe.x11-clocks ---
--- describe.x11-drivers ---
--- describe.x11-fm ---
--- describe.x11-fonts ---
--- describe.x11-servers ---
--- describe.x11-themes ---
--- describe.x11-toolkits ---
--- describe.x11-wm ---
 Done../compare-index.sh: INDEX /jails/index/usr/ports/INDEX-15 md5 fd1ebf3802c177b36e14a7262bdeeda3, previously b3625a7dc7924e55bc9df2206e6acd5c
./compare-index.sh: INDEX changed, processing
index_pkgversions.py: 38881 rows, 35234 origins, 0 malformed lines
./compare-index.sh: 38881 packages read from the INDEX
./compare-index.sh: 38881 rows loaded and compared
0	/var/db/freshports/cache/spooling/refresh.txt
0	/var/db/freshports/cache/spooling/not-in-index.txt
0	/var/db/freshports/cache/spooling/not-in-freshports.txt
$

This is good.

If there were rows to update, then this gets run:

refresh-listed-ports.pl /var/db/freshports/cache/spooling/refresh.txt

I’ve set up a cronjob to run compare-index.sh nightly – let’s see how that goes and how long we have to wait to find a divergence. From the same session as the refresh job, I ran this:

$ cat /usr/local/etc/cron.d/compare-daily-index 
4 2 * * * freshports /usr/local/libexec/freshports/compare-index-daily.sh
$ /usr/local/libexec/freshports/compare-index-daily.sh
$

The log entries are:

Oct  1 12:14:08 dev-ingress01 ./compare-index.sh[38435]: starts
Oct  1 12:14:08 dev-ingress01 ./compare-index.sh[38435]: fetching into /jails/index/usr/ports
Oct  1 12:14:09 dev-ingress01 ./compare-index.sh[38435]: checking /jails/index/usr/ports out at 40028bd01d75f6b9c896f416bc23ea22bf06f7e2
Oct  1 12:14:09 dev-ingress01 ./compare-index.sh[38435]: building the INDEX in jail index
Oct  1 12:26:25 dev-ingress01 ./compare-index.sh[38435]: INDEX /jails/index/usr/ports/INDEX-15 md5 fbf42617ce25b654735abad5acecb25a, previously 998e42bb0b24ddd50100a3d7423a9d1d
Oct  1 12:26:25 dev-ingress01 ./compare-index.sh[38435]: INDEX changed, processing
Oct  1 12:26:25 dev-ingress01 ./compare-index.sh[38435]: 38812 packages read from the INDEX
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: the comparison took 2s
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: 38812 rows loaded and compared
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: refresh: devel/p5-Moose: FreshPorts has 2.4000,1, INDEX has 2.4001,1
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: refresh: www/drupal11: FreshPorts has 11.4.5, INDEX has 11.4.8
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: refresh: www/elgg: FreshPorts has 7.0.5, INDEX has 7.1.0
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: refresh: www/moodle45: FreshPorts has 4.5.13, INDEX has 4.5.14
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: refresh: www/moodle50: FreshPorts has 5.0.9, INDEX has 5.0.10
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: refresh: www/moodle51: FreshPorts has 5.1.6, INDEX has 5.1.7
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: refresh: www/moodle52: FreshPorts has 5.2.2, INDEX has 5.2.3
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: refresh: 7 ports
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: index-behind: 0 ports
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: not-in-index: 0 ports
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: not-in-freshports: 0 ports
Oct  1 12:26:27 dev-ingress01 ./compare-index.sh[38435]: ends

Eh? Why so many? Ahh, commit processing was halted. There are commits waiting to process.

[12:40 dev-ingress01 dvl ~] % ls ~ingress/message-queues/incoming 
2026.10.01.11.50.52.000000.0f9467f2e35903bf866396c45b9743a3120f960c.xml  2026.10.01.11.50.53.000004.8ea4253d27cc3642a066433b83526c3633fa1d56.xml
2026.10.01.11.50.52.000001.13ad72fe8b816105aa8472802e5344f272060a14.xml  2026.10.01.11.50.53.000005.c9589fdf9b875c2b77ec0b47ab22563e23417d64.xml
2026.10.01.11.50.53.000002.a890fd401ecab7ee141800d1be08210ccc95fc98.xml  2026.10.01.11.59.05.000000.40028bd01d75f6b9c896f416bc23ea22bf06f7e2.xml
2026.10.01.11.50.53.000003.06c843a9e3dd8ca71c4df8ec1a91cf287f163c61.xml  2026.10.01.12.31.42.000000.37c9eba1644b0f9e1b5d130ccff381122f48aebe.xml

The repo at /jails/freshports/usr/ports is current with all commits (i.e. a git pull has been done). The commits just haven’t been process from the above XML files into the database.

Let’s get those commits processed and handled. I forgot how I should be doing this. The git repo in the index jail should be current to the latest commit processed by FreshPorts. INDEX should be built from that data.

To get back on track, I enabled commit processing and waited for the queue to clear:

[12:41 dev-ingress01 dvl ~] % sudo service freshports status                
freshports is not running.
[12:45 dev-ingress01 dvl ~] % sudo service freshports start 
Starting freshports.
[12:45 dev-ingress01 dvl ~] % ls ~ingress/message-queues/incoming      
[12:47 dev-ingress01 dvl ~] % sudo service freshports stop       
Stopping freshports.
Waiting for PIDS: 62257.
[12:47 dev-ingress01 dvl ~] %

There are no errors:

[12:47 dev-ingress01 dvl ~] % ls ~freshports/message-queues/recent/*.errors
zsh: no matches found: /var/db/freshports/message-queues/recent/*.errors

Let’s try again:

$ /usr/local/libexec/freshports/compare-index-daily.sh

Now that that is running, checking the trees gives consistent results:

[12:48 dev-ingress01 dvl ~] % sudo git -C /jails/index/usr/ports status    
HEAD detached at 40028bd01d75
nothing to commit, working tree clean
[12:49 dev-ingress01 dvl ~] % sudo git -C /jails/freshports/usr/ports status
HEAD detached at 40028bd01d75
nothing to commit, working tree clean
[12:49 dev-ingress01 dvl ~] %

*time passes*

I see this in the, an unchanged INDEX which makes sense: no new commits in the repo (only in the DB):

Oct  1 12:49:04 dev-ingress01 ./compare-index.sh[93079]: starts
Oct  1 12:49:04 dev-ingress01 ./compare-index.sh[93079]: fetching into /jails/index/usr/ports
Oct  1 12:49:05 dev-ingress01 ./compare-index.sh[93079]: checking /jails/index/usr/ports out at 40028bd01d75f6b9c896f416bc23ea22bf06f7e2
Oct  1 12:49:05 dev-ingress01 ./compare-index.sh[93079]: building the INDEX in jail index
Oct  1 13:01:12 dev-ingress01 ./compare-index.sh[93079]: INDEX /jails/index/usr/ports/INDEX-15 md5 fbf42617ce25b654735abad5acecb25a, previously fbf42617ce25b654735abad5acecb25a
Oct  1 13:01:12 dev-ingress01 ./compare-index.sh[93079]: INDEX unchanged, not processing

The corresponding email notification was:

Subject: FreshPorts -- compare-index -- 40028bd01d75f6b9c896f416bc23ea22bf06f7e2
Date: Thu, 01 Oct 2026 13:01:12 +0000
Message-Id: <6abe5998.d62e2.6c4b7697@dev-ingress01.int.unixathome.org>
From: <freshports@dev-ingress01.int.unixathome.org>

./compare-index.sh: fetching into /jails/index/usr/ports
./compare-index.sh: checking /jails/index/usr/ports out at 40028bd01d75f6b9c896f416bc23ea22bf06f7e2
HEAD is now at 40028bd01d75 devel/p5-Moose: Update to 2.4001
./compare-index.sh: building the INDEX in jail index
Generating INDEX-15 - please wait..
 Done../compare-index.sh: INDEX /jails/index/usr/ports/INDEX-15 md5 fbf42617ce25b654735abad5acecb25a, previously fbf42617ce25b654735abad5acecb25a
./compare-index.sh: INDEX unchanged, not processing

I think we’re good to go.

Next step

The next step: after compare-index.sh completes, run ./refresh-listed-ports.pl /var/db/freshports/cache/spooling/refresh.txt.

For now, it’s time to get to work.

Later that day…

Yes, remember to restart stopped services:

[15:32 dev-ingress01 dvl ~] % sudo service freshports status                
freshports is not running.
[15:56 dev-ingress01 dvl ~] % sudo service freshports start 
Starting freshports.
[15:56 dev-ingress01 dvl ~] % ls ~ingress/message-queues/incoming          
2026.10.01.13.28.06.000000.449676989847f2445776c7b95f2a6960b4a8bb15.xml
2026.10.01.13.35.45.000000.cc074534a3bcd21ee156e350bb8428b67f4f8dac.xml
2026.10.01.13.35.46.000001.a1bb9896a491134b523f443fa3b80da5b9612220.xml
2026.10.01.13.35.46.000002.de4aa8bc6f7e0be5d9a1e017dd8ea3387c828767.xml
2026.10.01.13.58.01.000000.7bc3a31c949968fb70a8cccf474abb38f59740a9.xml
2026.10.01.14.07.46.000000.c515a95b286e2be0c32a00784f3d51966a14968b.xml
2026.10.01.14.53.11.000000.830b1fc2b74b9f3d3ade109685a441ed7d21d3b1.xml
2026.10.01.15.04.48.000000.2748019034f036e7c18b0dcd7713d972b6e45eed.xml
2026.10.01.15.10.52.000000.3ff3fef65b87e731c8f4c793a994d139cbbab50d.xml
2026.10.01.15.11.29.000000.909e82014dfe1ba246300243d8c1fc9f7c6837f7.xml
[15:56 dev-ingress01 dvl ~] %
Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Related Posts

Leave a Comment

Scroll to Top