Dan Langille

I've been playing with Open Source software, starting with FreeBSD, since New Zealand Post installed DSL on my street in 1998. From there, I started writing at The FreeBSD Diary, moving my work here after I discovered WordPress. Along the way, I started the BSDCan and PGCon conferences. I slowly moved from software development into full time systems administration and now work for very-well known company who has been a big force in the security industry.

Oct 232021
 

Messages such as:

Internal error: I was expecting a short description and found nothing for databases/postgresql

indicate a cache entry which predates a change to the cache format. I could clear out the entire cache, but I want to try a targeted approach. In issue 339, some detail is provided. The common thread seems to be a deleted port.

My theory: at one time these database entries contains proper data. A script which updated all ports from the repo failed to detect a deleted port and information already in the database was lost during the update.

There is a table for recording items to be cleared from cache:

freshports.devgit=# \d cache_clearing_ports
                                        Table "public.cache_clearing_ports"
   Column   |            Type             | Collation | Nullable |                     Default                      
------------+-----------------------------+-----------+----------+--------------------------------------------------
 id         | integer                     |           | not null | nextval('cache_clearing_ports_id_seq'::regclass)
 port_id    | integer                     |           | not null | 
 category   | text                        |           | not null | 
 port       | text                        |           | not null | 
 date_added | timestamp without time zone |           | not null | now()
Indexes:
    "cache_clearing_ports_pkey" PRIMARY KEY, btree (id)
    "cache_clearing_ports_port_id_idx" UNIQUE CONSTRAINT, btree (port_id)
Foreign-key constraints:
    "cache_clearing_ports_port_id_fkey" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE CASCADE ON DELETE CASCADE

This populates the table with all deleted ports:

freshports.devgit=# begin;
BEGIN
freshports.devgit=# insert into cache_clearing_ports (port_id, category, port)  
select id, (select name from categories where id = category_id), (select name from element where id = element_id) from ports where status = 'D';
INSERT 0 20715
freshports.devgit=# select count(*) from cache_clearing_ports;
 count 
-------
 20715
(1 row)

freshports.devgit=# commit;
COMMIT

This command notifies the application which is listening on the front end:

freshports.devgit=# NOTIFY port_updated;
NOTIFY

In the logs of that application, I see:

Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: Just woke up! *************
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: Got NOTIFY: pid='33939', channel='port_updated', payload=''
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: found key port_updated
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: invoking RemoveCacheEntry()
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: checking for cache entries to remove...
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: COUNT: 20715 entries to process
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: removing glob /var/db/freshports/cache/ports/databases/phpmyadmin-devel/*
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: removing glob /var/db/freshports/cache/categories/databases/*
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: DELETE FROM cache_clearing_ports WHERE id = 32289779

It eventually ends around:

Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: DELETE FROM cache_clearing_ports WHERE id = 32310493
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: finished
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: invoking ClearMiscCaches()
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: invoked: ClearMiscCaches()
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: ClearMiscCaches() is clearing out entries in /var/db/freshports/cache/news/
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: ClearMiscCaches() is removing /var/db/freshports/cache/news/index-html-head-pagesize100-pagenum1-html
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: finished: ClearMiscCaches()

That is about 15 minutes to clear out 20715 entries, roughly 1400 a minute or about 22 per second.

It is also doing one transactions per cache-clearing.

Oct 142021
 

Recently I started seeing this message in the production logs for FreshPorts:

Oct 13 15:35:38 aws-1-nginx01 FreshPorts[75290]: _pkgmessage_UCL found a type is it not prepared for : Array

What code produces that message?

GitHub makes searching for that message easy. I found it here:

syslog(LOG_ERR, '_pkgmessage_UCL found a type is it not prepared for : ' . $thing->type);

What port has this issue?

Searching the webserver logs, I find:

$ grep 15:35:38 /var/log/nginx/freshports.org-access.log.0
92.220.10.100 - - [13/Oct/2021:15:35:38 +0000] "GET /graphics/rawtherapee/?branch=2021Q3 
HTTP/1.1" 200 13068 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"

Always be able to reproduce the issue

If I cannot reproduce the issue myself, I cannot be confident that any changes I make have fixed it.

After clearing the cache for that port I was able to reproduce the error message:

$ sudo rm -rf ~freshports/cache/ports/graphics/rawtherapee

Analysis of the problem

Looking at the code I can see it is saving the pkgmessage column (from the ports table) into a file. It then converts that to JSON and parses the output to create HTML.

The code is prepared to find various action values, such as upgrade, install, and remove.

But it’s getting an array.

Duplicating the issue from the command line

Let’s look at what ucl is bringing back. This is where some command line use of psql helps. I will abbreviate the output to reduce your scrolling.

freshports.devgit=# select pkgmessage from ports_active where name = 'rawtherapee';
                               pkgmessage                                
-------------------------------------------------------------------------
 [                                                                      +
 {type: [install, upgrade], message=<<EOD                               +
 LENSFUN INFORMATION:                                                   +
                                                                        +
 This package uses lensfun to correct lens aberrations. In case         +
 your camera or lens seem unsupported, try running                      +
 lensfun-update-data - this will download new lensfun databases.        +
                                                                        +
....

 remove older cache directories.                                        +
                                                                        +
 Also, after configurations have been moved to the new version's        +
 directory, older $HOME/.config/RawTherapee* directories may be removed.+
 EOD                                                                    +
 },                                                                     +
 ]
(5 rows)

I can not use that output in a file and run it through UCL. I need to get ride of the column header (—) and the field separators (+++).

I read the output of psql –help and came up with this:

[dan@pg02:~]: $ psql --no-align --tuples-only freshports.devgit -c "select pkgmessage from ports_active where name = 'rawtherapee'"
[
{type: [install, upgrade], message=<<EOD
LENSFUN INFORMATION:

This package uses lensfun to correct lens aberrations. In case
your camera or lens seem unsupported, try running
lensfun-update-data - this will download new lensfun databases.

DISK SPACE WARNING:

...

Also, after configurations have been moved to the new version's
directory, older $HOME/.config/RawTherapee* directories may be removed.
EOD
},
]

That is much better.

Let’s feed that into a file:

[dan@pg02:~]: $ psql --no-align --tuples-only freshports.devgit -c "select pkgmessage from ports_active where name = 'rawtherapee'" > rawtherapee.ucl

Now run ucl on it (the output has been wrapped to ease reading):

[dan@pg02:~]: $ /usr/local/bin/ucl_tool --in rawtherapee.ucl --format json
[
    {
        "type": [
            "install",
            "upgrade"
        ],
        "message": "LENSFUN INFORMATION:\n\nThis package uses lensfun to correct lens 
aberrations. In case\nyour camera or lens seem unsupported, try running\nlensfun-update-data - 
this will download new lensfun databases.\n\nDISK SPACE WARNING:\n\nNote that RawTherapee uses 
version-dependent cache and configuration\ndirectories.  Please be advised that cache 
directories can grow large,\nso be sure to check all users' $HOME/.cache/RawTherapee* and have 
them\nremove older cache directories.\n\nAlso, after configurations have been moved to the new 
version's\ndirectory, older $HOME/.config/RawTherapee* directories may be removed."
    }
]

There it is. type is an array in this case. The code always assumes it is a string.

Now it’s just a matter of adjusting the code to cater for this.

Sep 272021
 

This is the first of a series of posts which provide sample queries. Sometimes you need a starting point. This particular query will be used for showing the recent commits on a branch.

WITH branch_commits AS (
  SELECT distinct CL.id, CL.commit_DATE
    FROM commit_log CL
    JOIN commit_log_ports CLP on CL.id = CLP.commit_log_id
    JOIN commit_log_branches CLB ON CLP.commit_log_id = CLB.commit_log_id
    JOIN system_branch        SB ON SB.id = CLB.branch_id AND SB.branch_name = '2021Q3'
    ORDER BY CL.commit_DATE DESC
    LIMIT 100
)
SELECT CL.id, CL.commit_date, CL.message_id
  FROM branch_commits BC JOIN commit_log CL ON BC.id = CL.id
  ORDER BY CL.commit_date DESC;
Aug 252021
 

These are the steps I’m following to add the ports_to_refresh table.

This is a list of issues I will track through this upgrade.

  1. branch badge not populated for commits on branch #327
  2. base/binutils page broken #328
  3. Wrong info about hplip-plugin port when using search #322
  4. Package message not presented for x11/nvidia-hybrid-graphics/files/pkg-message.in … #321
  5. 404: Warning: htmlentities() expects parameter 1 to be string, resource given in /usr/local/www/freshports/rewrite/missing.php on line 38 #329
  6. port.pm has an if which can be deleted #325
  7. date.php has in correct links to previous and next dates #323
  8. Direction to use nonexistent path /var/db/repos/ports/distfiles #319
  9. Adjust color scheme (in dark mode?) #330
  10. distinfo sections are sometimes too wide #320

These are the steps to upgrade a node:

Run these items to update the database via psql:

\i datatype.txt
\i sp.txt
\i updates-2021-08-24-ports-to-refresh.ddl

On the website node:

sudo pkg upgrade

Restart the website node.

On the ingress node:

sudo service freshports stop
sudo service ingress stop
sudo sysrc freshports_enable="NO"
sudo sysrc ingress_enable="NO"
sudo pkg upgrade

Restart the ingress node.

cd /usr/local/etc/freshports
sudoedit config.pm config.pm.sample

Search for FreshPorts::Config::PortsToRefresh and copy it over from the sample file.

Look for diffs:

sudo diff -ruN config.pm config.pm.sample | less

Copy over changes:

sudoedit config.sh config.sh.sample

Check diffs:

sudo diff -ruN config.sh config.sh.sample | less
sudo sysrc ingress_enable="YES"
sudo service ingress start

Wait for the first commit to come in, then:

sudo service ingress stop
sudo sysrc freshports_enable="YES"
cd ~ingress/message-queues/incoming/
sudo mv -i * ../holding
cd ../holding

Move one .xml file over to ../incoming

Then start processing:

sudo service freshports start

Monitor the commit processing.

to be done: update for the other issues above.

For nrpe, the following source file needs to be refreshed on the ingress host:

roles/nrpe/templates/nrpe-sets/freshports-ingress.j2

Invoke this command to do the refresh:

ansible-playbook nrpe.yml --tags=nrpe_sets --limit=test-ingress01.int.unixathome.org

This is the required result:

[dan@test-ingress01:~] $ grep -r check_freshports_ports_to_refresh /usr/local/etc/nrpe.d/freshports-ingress.cfg
command[check_freshports_ports_to_refresh] = /usr/local/libexec/nagios-custom/check_freshports_ports_to_refresh

When ready to test the ports_to_refresh feature (the table is pre-populated with base/binutils by the DDL file run earlier), issue this command:

echo touch ~freshports/signals/ports_to_refresh ~freshports/signals/job_waiting | sudo su -fm freshports

validation

base/binutils

Verify that base/binutils on this node resembles:

badges on commits

Confirm that a badge is shown here:

https://test.freshports.org/commit.php?category=www&port=drupal7&files=yes&message_id=12184f4ef0a9565dcc8b8cac238ed34deb19b071 (adjust the hostname for the node you are working on. See https://github.com/FreshPorts/freshports/issues/327

P.license_restricted ports

re https://github.com/FreshPorts/freshports/issues/322

Issue this fix into postgresql:

freshports.devgit=# begin; insert into cache_clearing_ports (port_id, category, port) select P.id as port_id,
       C.name as category,
       E.name as port
  FROM ports P join categories C on p.category_id = C.id
               join element    E on p.element_id = E.id
 WHERE P.license_restricted is not null;

commit;

notify port_updated;

Monitor sudo tail -F /var/log/freshports/freshports.log while issuing the above. Much cache clearing should occur.

Then view https://www.freshports.org/search.php?query=hplip for the node in question.

pkg-message

re https://github.com/FreshPorts/freshports/issues/322

To refresh all nodes:

freshports.testgit=# begin;
BEGIN
freshports.testgit=# 
freshports.testgit=# insert into ports_to_refresh(port_id) select id from ports_active;
INSERT 0 46220
freshports.testgit=# 

Issue a commit.

Then on the ingress node:

echo touch ~freshports/signals/ports_to_refresh ~freshports/signals/job_waiting | sudo su -fm freshports

This update may take a while.

HTMEntities

re https://github.com/FreshPorts/freshports/issues/329

Visit https://test.freshports.org/audio/multimedia/ for the correct node and be logged in.

port.pm

re https://github.com/FreshPorts/freshports/issues/325

Nothing to check here, if things are working, it’s working.

date.php

re https://github.com/FreshPorts/freshports/issues/323

visit https://test.freshports.org/date.php?date=2021/7/24 for the correct node and be able to scroll through dates

nonexistant directories

re: https://github.com/FreshPorts/freshports/issues/319

This should already be handled by previous refreshes:

freshports.testgit=# begin;
BEGIN
freshports.testgit=# insert into ports_to_refresh(port_id) select id from ports_active where category = 'net' and name = 'citrix_ica';
INSERT 0 1
freshports.testgit=# commit;
COMMIT
freshports.testgit=# 

Then on the ingress node:

echo touch ~freshports/signals/ports_to_refresh ~freshports/signals/job_waiting | sudo su -fm freshports

Visit https://dev.freshports.org/net/citrix_ica/ for the correct node and see a reference /usr/ports.

dark mode colors

re https://github.com/FreshPorts/freshports/issues/330

wide distinfo

re https://github.com/FreshPorts/freshports/issues/320

see https://test.freshports.org/editors/aee#distinfo on the correct node.

Aug 142021
 

This post is the latest in a series of posts documenting the process of converting from using a chroot to using a full proper jail.

I spent about 2 hours of this fine Saturday morning writing this up and carrying out the steps.

As a result, both dev and test are now using a FreeBSD jail to extract data from ports in order to populate the database.

These are working notes, not so much a tutorial. However, you might find it useful too. I know I will when it comes to to convert stage tomorrow.

When you see mkjail below, that’s from sysutils/mkjail.

In this post:

  • on the jail – means the parent jail
  • on the child jail – means the child jail
  • on the host – means the server, the main server, hosting all the jails
    1. These are the steps used to update test-ingress01 from using ~freshports/ports-jail to /jails/freshports
      2021-08-14
      
      
      on the host:
      
      sudo zfs snapshot system/jails/test-ingress01@before-moving-to-jail-freshports
      
      on the jail:
      
      sudo service freshports stop
      sudo service ingress stop
      
      sudo sysrc freshports_enable="NO"
      sudo sysrc ingress_enable="NO"
      
      
      on the host:
      
      sudo service jail stop test-ingress01
      
      sudo mv /etc/fstab.test-ingress01 /etc/fstab.test-ingress01.NOT-IN-USE
      
      
      sudo zfs create -o mountpoint=none nvd/freshports/jailed/test-ingress01
      sudo zfs create -o mountpoint=none nvd/freshports/jailed/test-ingress01/jails
      sudo zfs create -o mountpoint=none nvd/freshports/jailed/test-ingress01/mkjail
      
      
      sudoedit /etc/jail.conf
      
      Add this to test-ingress01 jail definition:
      
          allow.chflags;
      
          # added when trying to get devfs in subjails
          allow.mount.fdescfs;
      
          allow.mount;
          allow.mount.devfs;
          allow.mount.linprocfs;
          allow.mount.nullfs;
          allow.mount.procfs;
          allow.mount.tmpfs = 1;
          allow.mount.zfs;
      
          allow.raw_sockets;
          allow.socket_af;
      
          children.max=6;
      
          exec.created+="zfs set jailed=on nvd/freshports/jailed/test-ingress01";
          exec.created+="zfs jail $name    nvd/freshports/jailed/test-ingress01";
      
          exec.poststart  += "jail -m allow.mount.linprocfs=1 name=$name";
      
          enforce_statfs=1;
      
      EDIT: 2021-09-09 - enforce_statfs was missing from the above - re https://twitter.com/DLangille/status/1435965521959215109
      
      
      Start the jail:
      
      sudo service jail start test-ingress01
      
      
      in the jail:
      
      verify zfs
      
      [dan@test-ingress01:~] $ zfs list
      [dan@test-ingress01:~] $ zfs list
      NAME                                          USED  AVAIL  REFER  MOUNTPOINT
      nvd                                          81.7G   133G    23K  none
      nvd/freshports                               81.5G   133G    23K  none
      nvd/freshports/jailed                        2.48G   133G    24K  none
      nvd/freshports/jailed/test-ingress01           72K   133G    24K  none
      nvd/freshports/jailed/test-ingress01/jails     24K   133G    24K  none
      nvd/freshports/jailed/test-ingress01/mkjail    24K   133G    24K  none
      [dan@test-ingress01:~] $ 
      
      
      Set mount points:
      
      sudo zfs set mountpoint=/jails         nvd/freshports/jailed/test-ingress01/jails
      sudo zfs set mountpoint=/var/db/mkjail nvd/freshports/jailed/test-ingress01/mkjail
      
      
      [dan@test-ingress01:~] $ zfs list
      NAME                                          USED  AVAIL  REFER  MOUNTPOINT
      nvd                                          81.7G   133G    23K  none
      nvd/freshports                               81.5G   133G    23K  none
      nvd/freshports/jailed                        2.48G   133G    24K  none
      nvd/freshports/jailed/test-ingress01           72K   133G    24K  none
      nvd/freshports/jailed/test-ingress01/jails     24K   133G    24K  /jails
      nvd/freshports/jailed/test-ingress01/mkjail    24K   133G    24K  /var/db/mkjail
      [dan@test-ingress01:~] $ 
      
      
      
      Creating the jail within a jail:
      
      sudo pkg install mkjail
      
      sudoedit /usr/local/etc/mkjail.conf :
      
      ZPOOL="nvd"
      JAILDATASET="freshports/jailed/test-ingress01/jails"
      
      sudo mkjail create -a amd64 -j freshports -v 12.2-RELEASE
      
      If that fails with:
      
      tar: could not chdir to '/12.2-RELEASE/'
      
      Then run the same command again.
      
      sudo sysrc jail_enable="YES"
      
      su to root, enter a bash shell, and run this:
      
      cat << EOF > /etc/jail.conf
      exec.start = "/bin/sh /etc/rc";
      exec.stop = "/bin/sh /etc/rc.shutdown";
      exec.clean;
      mount.devfs;
      path = /jails/\$name;
      allow.raw_sockets;
      securelevel = 2;
      exec.consolelog="/var/tmp/jail-\$name";
      
      host.hostname = "\$name\$(hostname)";
      
      persist;
      freshports {
          host.hostname = "freshports";
      
          ip4 = inherit;
          persist;
      
          devfs_ruleset=0;
      
          allow.mount=true;
          enforce_statfs=1;
          allow.mount.devfs;
          allow.mount.procfs;
      }
      EOF
      
      mkdir -p /jails/freshports/usr/local/etc/pkg/repos
      echo "FreeBSD: { enabled: no }" > /jails/freshports/usr/local/etc/pkg/repos/FreeBSD.conf
      
      cat << EOF > /jails/freshports/usr/local/etc/pkg/repos/local.conf
      local: {
         url: "pkg+http://fedex.unixathome.org/packages/122amd64-default-master-list/"
         mirror_type: "srv",
         signature_type: "PUBKEY",
         pubkey: "/etc/ssl/slocum.unixathome.org.cert",   
         enabled: true
      }
      EOF
      
      
      cat << EOF > /jails/freshports/etc/ssl/slocum.unixathome.org.cert
      -----BEGIN PUBLIC KEY-----
      MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuveHTXwrwGmdWG6oFWgN
      R/7bOQiQIE9iFmXmy4/MX9+01zMh2mjTfIOFe8IE1QqOK7X5EkVqhIdHCFg1Cg/x
      Gl5oxxQEhp/HQbtVN7pNcpXKIGl0t5sFcjwXQqXS1wb87JP+v6KuOSTCFzS3l/X2
      XtOo5QJZtxuF7+IM2PZlYb8MDmhVxriPw0pWRiP8lyW2LV1dSrN+VFpllZYinfIv
      Sr7sUArIVlHH+Ddrm5MWjqTsHid36og26NDmAQnfX96IBF1sXadSTxKy3YwCnEcv
      L7mBJhNyTIULuSbknM9zP9amkrlyLhWl+SdRGRkcOmXwHgzbdZsL62OlkXYIgkB+
      tK099ARziCSe+sclhgfjoixnXxk0h9gUU6h5BDafATvtP4KDmwDYQEXO/7OPS0/H
      vHFLEWLExcbW6hF0fyy2aA/HTlX83bBqJ+evsFtcvyxNfDp7tnjus/oAmJ82IU4F
      0ZmWMoJDeNznO+3iokHH0J8vxa4kd1hjoSaDh1+qOZCXGRcsqyDTgWQEdQ5Uy76j
      c9NgCJqHqJyjPqsiIZmPJNGUh1f7VaSUT8a131X5dwj6kx02s55UGJeIlK6F1d1f
      7+7pdfv4rzHK3iPlP1eXQlv8szGAxdDbkSFqLK6gIC8V/Mf0B0zN0aU1JZkaAvoH
      GopjN8IyF6fx/5yN9EAp8GUCAwEAAQ==
      -----END PUBLIC KEY-----
      EOF
      
      
      cat << EOF > /jails/freshports/etc/resolv.conf
      search unixathome.org int.unixathome.org
      nameserver 10.55.0.1 
      nameserver 10.55.0.73
      nameserver 10.55.0.13
      EOF
      
      
      cat << EOF > /jails/freshports/etc/rc.conf
      cron_enable="NO"
      syslogd_enable="NO"
      sendmail_enable="NO"
      sendmail_submit_enable="NO"
      sendmail_outbound_enable="NO"
      sendmail_msp_queue_enable="NO"
      EOF
      
      
      service jail start
      
      jexec freshports
      
      Nothing but this should be running:
      
      root@freshports:/ # ps auwwx
      USER  PID %CPU %MEM   VSZ  RSS TT  STAT STARTED    TIME COMMAND
      root 8106  0.0  0.0 13196 3800  5  SJ   14:11   0:00.01 /bin/csh -i
      root 8376  0.0  0.0 11768 2776  5  R+J  14:12   0:00.00 ps auwwx
      
      pkg install pkg
      
      exit
      
      Now, back in the parent jail
      
      cd /jails/freshports/usr
      sudo git clone https://git.FreeBSD.org/ports.git
      
      
      
      sudo mv ~freshports/ports-jail ~freshports/ports-jail.NO.LONGER.IN.USE
      
      sudo pkg upgrade
      
      cd /usr/local/etc/freshports
      
      reconcile the configuration files:
      
      sudoedit config.ini config.ini.sample
      
      sudoedit config.pm config.pm.sample
      sudo diff -ruN config.pm config.pm.sample | less
      
      sudoedit config.sh config.sh.sample
      sudo diff -ruN config.sh config.sh.sample  | less
      
      
      On ansible:
      
      ansible-playbook freshports-ingress-git.yml --limit=test-ingress01.int.unixathome.org --tags=sudoers
      
      
      on the parent jail:
      
      sudo sysrc ingress_enable="YES"
      sudo service ingress start
      
      Watch the logs, wait for some commits.
      
      sudo service ingress stop
      
      cd /var/db/ingress/message-queues/incoming
      sudo mv -i * ../holding/
      
      sudo sysrc freshports_enable="YES"
      sudo service freshports start
      cd ../holding/
      
      Move one of the xml files into incoming:
      
      sudo mv -i BLAH.xml ../incoming/
      
      check for errors, repeat until confidence is high.
      
Jul 282021
 

I was asked: Why does FreshPorts have links to both git and subversion repos?

The short answer is: because at the time, the commits were committed to those respective repos. The links to the other repo do not exist.

FreeBSD moved from using subversion to git.

During the transition, the subversion repo became read-only. All the commits before that date, point to the subversion repo. All the commits after that date, point to the git repo. In the code, there is more to it than a date, but that’s the basic generalization.

Take, for example, my port sysutils/anvil, all the commit before 06 Apr 2021 have a link to the git repo (https://cgit.freebsd.org/ports/), and all the commits before that, have a link to the subversion repo (https://svnweb.freebsd.org/ports/).

There is no need to change, from what I can see. However, there may be a way to link all older commits to git.

For example, can a link such as https://svnweb.freebsd.org/ports?view=revision&revision=554836 be easily converted into a link to the git repo?

Short answer: no.

Long answer: maybe.

From a discussion on #gitcvt:

[Jul 27 10:25] <@dvl> How can I locate svn revisions in git? e.g. 563183 -> https://svnweb.freebsd.org/ports?view=revision&revision=563183
[Jul 27 10:26] <@dvl> I’m hoping for a https://cgit.freebsd.org/ports/commit/?id= type URL
[Jul 27 10:28] <@dvl> See examples at https://www.freshports.org/x11-fonts/jetbrains-mono/#history

[Jul 27 15:44] <fbsdslack> <imp> dvl: find an answer yet?
[Jul 27 15:44] <fbsdslack> <imp> I have a git command recipe
[Jul 27 15:45] <fbsdslack> <imp> is that of interest?

[Jul 27 16:54] <fbsdslack> <imp> % git clone -o freebsd –config remote.freebsd.fetch=‘+refs/notes/*:refs/notes/*’ https://git.freebsd.org/${repo}.git
[Jul 27 16:55] <fbsdslack> <imp> grabs the notes, and then you can use –grep ‘revision=563183’
[Jul 27 16:55] <fbsdslack> <imp> with git log to find the commit.
[Jul 27 16:56] <fbsdslack> <imp> Details are in the committer’s guide in “Finding the Subversion Revision” section (there’s no anchor for this)
[Jul 27 16:56] <fbsdslack> <imp> % git config –add remote.freebsd.fetch “+refs/notes/*:refs/notes/*”
[Jul 27 16:57] <fbsdslack> <imp> may be needed if you already have a free (assuming your origin is ‘freebsd’)

In addition, there may be a way to redirect from subversion to git, based on the revision number.

[Jul 27 15:48] <@lwhsu> dvl: I’m thinking to generate a map file from svn revision to git hash, and use that file to send http redirection to cgit.f.o on svnweb.f.o
[Jul 27 15:49] <@lwhsu> to generate the map file, I think use the git note in each repository is a way.

My reading of the above: I can’t just provide a subversion revision number and create a link with that. I could run a one-off script, using the notes, and find a link for all the subversion commits.

For now, I’d rather not do that, given that the subversion repo is still online. Eventually, though, as imp pointed out: For FreshPorts, you can run a relatively simple script to scrape all that once and then have it forever because the mappings at this point will never change and there will be no new svn r numbers in the ports tree.

Jul 042021
 

Right now, we have have:

  • dev.freshports.org
  • devgit.freshports.org
  • test.freshports.org
  • testgit.freshports.org
  • stage.freshports.org
  • stagegit.freshports.org

The goal: three hosts: dev, test, stage.

I’ll delete the existing: dev, test, stage

I’ll rename:

  • devgit -> dev
  • testgit -> test
  • stagegit -> stage

I think this will be easier than moving content from one jail to another.

Each of the three renamed hosts will have redirects to the new hosts.

EDIT: 2021-07-05 This work has been completed.

Jul 042021
 

Ports come and go. Sometimes they come back. Take a recent commit on 1 July against editors/ved (and others). This port was created on 01 Jan 2001 and removed on 15 May 2014 because it used the deprecated smake. The 1 July 2021 commit brought it back. There was a problem.

Here is the page after the commit:

FreshPorts page for editors/ved

FreshPorts page for editors/ved

The history shows that the version did not change with this commit.

FreshPorts history for editors/ved showing 1.71_1 for most recent commit and the previous commit

FreshPorts history for editors/ved showing 1.71_1 for most recent commit and the previous commit

If you go back into the 11 Apr 2014 commit for editors/ved and click on the Subversion link, you’ll wind up at revision 350895, you can see the previous Makefile.

With that 1 July commit, editors/ved became a secondary port and obtains its PORTVERSION information from the primary port devel/schilybase.

Looking at devel/schilybase, its version is 2021.06.07. The maintainer brought this issue to my attention.

Logs

In the logs I found:

[dan@devgit-ingress01:~/modules] $ cd ~freshports/message-queues/archive/2021_07/2021_07_01/
[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ less *2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3*.log
...
# # # # Resurrecting deleted ports # # # #
...
found a deleted port: port='ved', port_id='4397', element_id='48745'
...
  inspecting: 'Add' , '/ports/head/editors/ved/Makefile' , 'editors/ved/Makefile' , '2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3' , 'editors' , 'ved' , 'Makefile'
  ...found a file from that port
  ...hmmm, we are modifying a file for a port which is deleted...  ........ I will resurrect that port for you
Port editors/ved needs to be Resurrected
  ........ resurrection done!
  we will not examine any more files for this port!
  finished looking through the files
# # # # Finished resurrecting deleted ports # # # #

That last line is line 5937 of the log file.

That’s good. The code correctly detected that this commit was touching a dead port and brought it back to life. In the background, this was flipping a flag in the database from INACTIVE to ACTIVE.

But wait, there’s more, around line 7558 of this 14994 line file:

...
port = editors/ved, port_id = '4397', category_id='1', needs_refresh='7'
This port is deleted: not refreshing.
This port is deleted: not saving.
...

The code thinks editors/ved is deleted, and therefore does not refresh the database with the values it could extract from the Makefile.

What?

This is a left-hand right-hand situation where the code is reading a list of ports from the database at the start of the processing, modifying the database (setting the deleted port to be undeleted), and then not updating its local copy of that port.

It can be successfully argued that the code which does the deletion should be the setting that flag in the local copy. I think that is the way to go. However, for now, just for testing, I’m going to try this:

[dan@devgit-ingress01:~/modules] $ svn di
Index: verifyport.pm
===================================================================
--- verifyport.pm	(revision 5736)
+++ verifyport.pm	(working copy)
@@ -972,6 +972,8 @@
 				$extra         = $extra         // '';
 				print "  inspecting: '$action' , '$filename' , '$filename_stripped' , '$revision' , '$category_name' , '$port_name' , '$extra'\n";
 
+				# instead of doing this through $element, perhaps do it through $port
+				# when then invokes $element and then sets $port->{status}.
 				if ($category_name eq $port->{category} && $port->{name} eq $port_name) {
 					print "  ...found a file from that port\n";
 					if ($action eq $FreshPorts::Constants::ADD || $action eq $FreshPorts::Constants::MODIFY) {
@@ -983,6 +985,9 @@
 						$element->update_status();
 						print "  ........ resurrection done!\n";
 						print "  we will not examine any more files for this port!\n";
+
+						# For later, when we are updating ports, mark this port as active.
+						$port->{status} = $FreshPorts::Element::Active;
 						# we are looping through files for a single port, only resurrect once.
 						last;
 					}
[dan@devgit-ingress01:~/modules] $ 

Preparing the database for a test run

I’ll run this through the devgit database first. Grabbing the element id from the logs above, I ran this query:

freshports.devgit=# select *, element_pathname(id) from element where id = 48745;
  id   | name | parent_id | directory_file_flag | status |    element_pathname     
-------+------+-----------+---------------------+--------+-------------------------
 48745 | ved  |         2 | D                   | A      | /ports/head/editors/ved
(1 row)

freshports.devgit=# 

In order to ‘delete’, this port, I can run this query:

freshports.devgit=# begin;
BEGIN
freshports.devgit=# update element set status = 'D' where id = 48745;
UPDATE 1
freshports.devgit=# select *, element_pathname(id) from element where id = 48745;
  id   | name | parent_id | directory_file_flag | status |    element_pathname     
-------+------+-----------+---------------------+--------+-------------------------
 48745 | ved  |         2 | D                   | D      | /ports/head/editors/ved
(1 row)

freshports.devgit=# commit;
COMMIT
freshports.devgit=# 

Now when I view this port on devgit, I see:

editors/ved, now deleted (see the RIP tombstone)

editors/ved, now deleted (see the RIP tombstone)

The test run

This is how I rerun the commit. First, it is deleted from the database:

freshports.devgit=# begin;
BEGIN
freshports.devgit=# delete from commit_log where message_id = '2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3';
DELETE 1
freshports.devgit=# commit;
COMMIT
freshports.devgit=# 

Then, it is injected into the processing stream:

[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ ls *2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3.xml
[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ sudo mv -i 2021.07.01.21.11.52.000000.2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3.xml ~ingress/message-queues/incoming/
[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ 

A few seconds later, we have these files:

[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ cd ~freshports/message-queues/recent/
[dan@devgit-ingress01:/var/db/freshports/message-queues/recent] $ ls -l *2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3*
-rw-r--r--  1 freshports  freshports  891981 Jul  4 18:06 2021.07.01.21.11.52.000000.2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3.log
-rw-rw-r--  1 freshports  freshports   11993 Jul  1 21:15 2021.07.01.21.11.52.000000.2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3.xml
[dan@devgit-ingress01:/var/db/freshports/message-queues/recent] $ 

The editors/ved port now looks like this:

editors/ved with the correct version: 2021.06.07

editors/ved with the correct version: 2021.06.07

That’s editors/ved with the correct version: 2021.06.07

Success!

The most time consuming part of this exercise was tracing through the code to ensure that the values used by one part of the code matched the other part (left-hand vs right-hand mentioned above).

The logs again

This time the log was about 15400 lines and contained this at about line 8527:

port = editors/ved, port_id = '4397', category_id='1', needs_refresh='7'
into RefreshFromFiles()
working with repo='ports'
working on CommitBranch='main'
with svn_revision=''
this port *** EITHER *** does not need a refresh *** OR *** we were told not to fetch
RepoType='git'
Repository='ports'
CommitBranch='main'
calling FreshPorts::Branches::GetPathToRepoForBranch
CommitBranch:   'main
REPODIR:        '/var/db/freshports/ports-jail/var/db/repos/ports'
REPODIR_CHROOT: '/var/db/repos/ports'
checking to make sure /var/db/freshports/ports-jail/var/db/repos/ports/editors/ved/Makefile exists
Phew.  It's here.  Moving on....
makecommand = /usr/local/bin/sudo /usr/sbin/chroot -u freshports /var/db/freshports/ports-jail /make-port.sh /var/db/repos/ports editors/ved 2>/tmp/FreshPorts.editors.ved.make-error.2021.7.4.18.6.22.59948
Result = 0
trying to get master sites.  Errors will be in '/tmp/FreshPorts.editors.ved.make-mastersites-error.2021.7.4.18.6.23.59948'
'/usr/local/bin/sudo /usr/sbin/chroot -u freshports /var/db/freshports/ports-jail /make-master-sites-all.sh /var/db/repos/ports editors/ved 2>/tmp/FreshPorts.editors.ved.make-mastersites-error.2021.7.4.18
trying to get showconfig.  Errors will be in '/tmp/FreshPorts.editors.ved.showconfig.2021.7.4.18.6.23.59948'
'/usr/local/bin/sudo /usr/sbin/chroot -u freshports /var/db/freshports/ports-jail /make-showconfig.sh /var/db/repos/ports editors/ved 2>/tmp/FreshPorts.editors.ved.showconfig.2021.7.4.18.6.23.59948'
$result='0'
$showconfig='===> The following configuration options are available for ved-2021.06.07:
     DOCS=on: Build and/or install documentation
===> Use 'make config' to modify these settings'
 portname                 = 'ved'
 packagename              = 'ved'
...

This time the logs show that the port was being refreshed and the database updated.

Updating the code

While reviewing the ports code, I saw:

[dan@devgit-ingress01:~/modules] $ grep sub port.pm
sub _initialize {
sub _GetValuesFromRow {
sub new {
sub save {
sub savePortTableOnly {
sub _save {
sub FetchByID {
sub FetchByPartialPathName {
sub _FetchElementIDByPartialPathName {
sub _ExtractValuesFromMakefile {
sub _Validate {
sub _GetDescrAndHomePage($) {
sub _GetFileContentsFromJail($) {
sub _GetRealPath($) {
sub RefreshFromFiles($;$;$;$;$;$) {
sub GetNeedsRefreshForNewPort {
sub IsActive {
sub IsDeleted {
sub SetActive {
sub SetDeleted {
sub IsValidDate($) {
sub upate_generate_plist {
sub update_package_flavors {
sub update_depends {
sub depends_type_long {
sub update_depends_helper {
sub _addMissingPORTSDIR {
sub CreatePortOnBranch {
[dan@devgit-ingress01:~/modules] $ 

See the SetDeleted function there on line 18? That’s much better than what I initially had in my test code.

Instead of this:

+						# For later, when we are updating ports, mark this port as active.
+						$port->{status} = $FreshPorts::Element::Active;

I now have this:

+						# For later, when we are updating ports, mark this port as active.
+						$port->SetActive();

It works and uses the proper function too. However, it does not undelete the port. It just sets a value in the local copy.

More work is required there. I think a $port->Undelete() function is called for.

The patch

This is the tested (on exactly one commit) patch:

[dan@devgit-ingress01:~/modules] $ svn di
Index: port.pm
===================================================================
--- port.pm	(revision 5713)
+++ port.pm	(working copy)
@@ -199,7 +199,7 @@
 	my $CommitBranch = shift;
 	my $FullSave     = shift;
 
-	print "into FreshPorts::Port::save\n";
+	print "into FreshPorts::Port::_save\n";
 
 	#
 	# to save, element_id and category_id must be valid
@@ -352,6 +352,26 @@
 	return $this->{id};
 }
 
+sub Undelete {
+	my $this = shift;
+	my $dbh  = shift;
+
+	print "into FreshPorts::Port::Undelete\n";
+	my $element  = FreshPorts::Element->new($dbh);
+
+	$element->{id}     = $this->{element_id};
+	$element->{status} = $FreshPorts::Element::Active;
+
+	$element->update_status();
+
+	# For later, when we are updating ports, mark this port as active.
+	$this->SetActive();
+
+	print "leaving FreshPorts::Port::Undelete\n";
+
+	return $this->{id};
+}
+
 sub FetchByID {
 	my $this	= shift;
 
Index: verifyport.pm
===================================================================
--- verifyport.pm	(revision 5736)
+++ verifyport.pm	(working copy)
@@ -952,7 +952,7 @@
 	#
 	print "# # # # Resurrecting deleted ports # # # #\n\n";
 	while (my ($portname, $port) = each %Ports) {
-		if ($port->{status} eq $FreshPorts::Element::Deleted) {
+		if ($port->IsDeleted()) {
 			print "found a deleted port: port='$port->{name}', port_id='$port->{id}', element_id='$port->{element_id}'\n";
 			print "now looking for files which were modified...\n";
 
@@ -972,6 +972,8 @@
 				$extra         = $extra         // '';
 				print "  inspecting: '$action' , '$filename' , '$filename_stripped' , '$revision' , '$category_name' , '$port_name' , '$extra'\n";
 
+				# instead of doing this through $element, perhaps do it through $port
+				# when then invokes $element and then sets $port->{status}.
 				if ($category_name eq $port->{category} && $port->{name} eq $port_name) {
 					print "  ...found a file from that port\n";
 					if ($action eq $FreshPorts::Constants::ADD || $action eq $FreshPorts::Constants::MODIFY) {
@@ -978,11 +980,11 @@
 						print "  ...hmmm, we are modifying a file for a port which is deleted...";
 						print "  ........ I will resurrect that port for you\n";
 						FreshPorts::Utilities::ReportError('notice', "Port $category_name/$port_name needs to be Resurrected", 0);
-						$element->{id}     = $port->{element_id};
-						$element->{status} = $FreshPorts::Element::Active;
-						$element->update_status();
+						$port->Undelete($dbh);
+
 						print "  ........ resurrection done!\n";
 						print "  we will not examine any more files for this port!\n";
+
 						# we are looping through files for a single port, only resurrect once.
 						last;
 					}
[dan@devgit-ingress01:~/modules] $ 
Jun 272021
 

This outlines the steps for putting the new git-delta.sh script into use. These steps will be required on each ingress node.

stop ingress commit processing

Look at the output of sudo tail /var/log/freshports/git.log and you should see something like this:

[dan@devgit-ingress01:~] $ sudo tail /var/log/freshports/git.log
2021.06.27 17:21:05 git-delta.sh Repodir is /var/db/ingress/repos/src
2021.06.27 17:21:05 git-delta.sh Running: /usr/local/bin/git pull:
Already up to date.
2021.06.27 17:21:06 git-delta.sh Done.
2021.06.27 17:21:06 git-delta.sh STARTPOINT = b762974cf4b9ea77f1decf4a6d829372f0a97f75
2021.06.27 17:21:06 git-delta.sh Running: /usr/local/bin/git rev-list b762974cf4b9ea77f1decf4a6d829372f0a97f75..HEAD
2021.06.27 17:21:06 git-delta.sh Done.
2021.06.27 17:21:06 git-delta.sh No commits were found
2021.06.27 17:21:06 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo src --path /var/db/ingress/repos/src --commit b762974cf4b9ea77f1decf4a6d829372f0a97f75 --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
2021.06.27 17:21:06 git-delta.sh Ending

Since it was 13:21 and not close to a multiple of 3 (this script runs every 3 minutes), I knew that I could stop the ingress daemon:

[dan@devgit-ingress01:~] $ sudo service ingress stop
Stopping ingress.

Add the tags

The existing configuration uses files to store the ‘last commit’. See here:

[dan@devgit-ingress01:/var/db/ingress/repos] $ ls -l
total 28
drwxr-xr-x   7 ingress  ingress  11 Jun  5 18:51 doc
-rw-r--r--   1 ingress  ingress  41 Jun 27 17:21 latest.doc
-rw-r--r--   1 ingress  ingress  41 Jun 27 17:21 latest.ports
-rw-r--r--   1 ingress  ingress  41 Jun 27 17:21 latest.src
drwxr-xr-x  70 ingress  ingress  81 Jun 27 14:51 ports
drwxr-xr-x  27 ingress  ingress  44 Jun 27 01:00 src
[dan@devgit-ingress01:/var/db/ingress/repos] $ 

This is the script I created for that:

#!/bin/sh

# The main branch is always origin/main - so far
refname='origin/main'

repos="doc ports src"
for repo in $repos
do
  tag=$(cat latest.$repo)
  cd $repo
  git tag -m "last known commit of " -f freshports/origin/main $tag
  cd -
done

Running that:

$ # The main branch is always origin/main - so far
$ refname='origin/main'
$ 
$ repos="doc ports src"
$ for repo in $repos
> do
>   tag=$(cat latest.$repo)
>   cd $repo
>   git tag -m "last known commit of " -f freshports/origin/main $tag
>   cd -
> done
/var/db/ingress/repos
/var/db/ingress/repos
/var/db/ingress/repos

Not entirely exciting.

This is the script to display the existing values:

#!/bin/sh

# The main branch is always origin/main - so far
refname='origin/main'

repos="doc ports src"
for repo in $repos
do
  tag=$(cat latest.$repo)
  cd $repo
  git rev-parse --verify freshports/$refname^{}
  cd -
done

Running that:

[dan@devgit-ingress01:/var/db/ingress/repos] $ /bin/sh
$ # The main branch is always origin/main - so far
$ refname='origin/main'
$ 
$ repos="doc ports src"
$ for repo in $repos
> do
>   tag=$(cat latest.$repo)
>   cd $repo
>   git rev-parse --verify freshports/$refname
>   cd ..
> done
048a537b787b9e858df140dfdc2122e3fb49c198
/var/db/ingress/repos
cc60e220aa4fd4c3b0f5eb74b8eb13bcf6f9c406
/var/db/ingress/repos
ca79c1fd81f2b2694a157df3beecb6fb48483900
/var/db/ingress/repos
$ cat latest.doc
4778fc33a57b1a7b9597868fa7fadef63a891edd
$ cat latest.ports
03ed448b501283f1e4d5dd0abb6847a91a32a79b
$ cat latest.src
b762974cf4b9ea77f1decf4a6d829372f0a97f75
$ 

That seems wrong. Why is the hash from the git rev-parse –verify not the same.


EDIT: 2021-07-09 The solution to seeing the actual commit hash is to change the line to:

git rev-parse --verify freshports/$refname^{}

That deferences the tag. The above code has since been updated with that change, for my convenience.

end EDIT.


I spent 30-45 minutes looking in that. Then I did this:

$ git show freshports/origin/main
tag freshports/origin/main
Tagger: Dan Langille <dan@langille.org>
Date:   Sun Jun 27 18:59:35 2021 +0000

last known commit of origin/main

commit 03ed448b501283f1e4d5dd0abb6847a91a32a79b (tag: freshports/origin/main)
Author: Rene Ladan <rene@FreeBSD.org>
Date:   Sun Jun 27 16:56:24 2021 +0200

    sysutils/bhyve-firmware: undeprecate CSM option
    
    The sysutils/uefi-edk2-bhyve port no longer needs Python 2.7

diff --git a/sysutils/bhyve-firmware/Makefile b/sysutils/bhyve-firmware/Makefile
index 1fb78f76049c..3c601a11f7ef 100644
--- a/sysutils/bhyve-firmware/Makefile
+++ b/sysutils/bhyve-firmware/Makefile
@@ -14,11 +14,4 @@ OPTIONS_DEFAULT=     CSM
 CSM_DESC=              Include firmware with Compatibility Support Module
 CSM_RUN_DEPENDS=       uefi-edk2-bhyve-csm>=0:sysutils/uefi-edk2-bhyve-csm
 
-.include <bsd.port.options.mk>

That commit hash matches the value in latest.ports. All is good. This has done the right thing.

Move the script into place

I copied the script from mydev to test-ingress01:

$ scp git-delta.sh devgit-ingress01:
git-delta.sh                                                                                                                                                              

Then moved it into to place:

[dan@devgit-ingress01:/var/db/ingress/repos] $ mv -i ~/git-delta.sh  /usr/local/libexec/freshports/git-delta.sh
[dan@devgit-ingress01:/var/db/ingress/repos] $ sudo pkg check -sr
Checking all packages:  87%
py38-freshports-git-proc-commit-0.0.4_1: checksum mismatch for /usr/local/libexec/freshports/git-delta.sh
Checking all packages: 100%
[dan@devgit-ingress01:/var/db/ingress/repos] $ ls -l /usr/local/libexec/freshports/git-delta.sh
-rwxr-xr-x  1 dan  dan  3977 Jun 27 18:57 /usr/local/libexec/freshports/git-delta.sh

Yep, that’s the right file.

I’m overwriting the file usually installed by package. Most of /usr/local/libexec/freshports/ is part of an svn co of the code. But a few files are not.

[dan@devgit-ingress01:/usr/local/libexec/freshports] $ svn st
?       .idea
?       UpdatePackagesFromRawPackages.py
?       check_freshports_cache_clearing
?       check_git.sh
?       check_report_logs.pl
?       check_repos_for_new_stuff.py
?       fetch-extract-parse-import-one-abi.sh
?       get_packagesite.txz_date
?       git-delta.sh
?       git-range-of-commits.sh
?       git-single-commit.sh
?       git-to-freshports-xml.py
?       helper_scripts
?       import-via-copy-packagesite-all-raw-fields.py
?       import_packagesite.py
?       show-config.sh
?       test-categories.pl
?       test-new-xml-code-single.sh
?       test-new-xml-code-starting-from.sh
?       testing.pm
[dan@devgit-ingress01:/usr/local/libexec/freshports] $ pkg info -l py38-freshports-git-proc-commit 
py38-freshports-git-proc-commit-0.0.4_1:
	/usr/local/libexec/freshports/check_git.sh
	/usr/local/libexec/freshports/git-delta.sh
	/usr/local/libexec/freshports/git-range-of-commits.sh
	/usr/local/libexec/freshports/git-single-commit.sh
	/usr/local/libexec/freshports/git-to-freshports-xml.py
[dan@devgit-ingress01:/usr/local/libexec/freshports] $ 

Now let’s wake up the sleeping ingress daemon.

Trying out the script

I’m going to run the ingress daemon after stopping the freshports daemon. I want to look for incoming commits and create the XML, but I don’t want to process the XML.

$ sudo service freshports stop
Stopping freshports.

Now start:

[dan@devgit-ingress01:~] $ sudo service ingress start
Starting ingress.

And the logs pile up:

[dan@devgit-ingress01:/usr/local/libexec/freshports] $ sudo xtail /var/log/freshports/*.log
^C
*** recently changed files ***
   1  27-Jun-21 19:36:09  /var/log/freshports/ingress-daemon.log
   2  27-Jun-21 19:36:09  /var/log/freshports/freshports.log
   3  27-Jun-21 17:21:06  /var/log/freshports/git.log
   4  27-Jun-21 17:15:11  /var/log/freshports/freshports-daemon.log
   5  27-Jun-21 03:04:23  /var/log/freshports/report-new-ports.daily.log
   6  27-Jun-21 03:04:22  /var/log/freshports/report-notification.daily.log
   7  27-Jun-21 03:03:01  /var/log/freshports/report-security.daily.log
   8  27-Jun-21 00:00:00  /var/log/freshports/report-notification.weekly.log
currently watching:  15 files  0 dirs  0 unknown entries

*** /var/log/freshports/ingress-daemon.log ***
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: starting up!
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: incoming: /var/db/ingress/message-queues/incoming
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: ready
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: yes, there is a job waiting
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: running /usr/local/bin/perl ./job-waiting.pl
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: from directory  /usr/local/libexec/freshports
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: -rw-r--r--  1 dan  dan  2421 May  9 12:36 ./job-waiting.pl

*** /var/log/freshports/freshports.log ***
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: running job-waiting.pl 
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: starting ./job-waiting.pl (/usr/local/libexec/freshports) 
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: running ./job-waiting.pl as user = 'ingress' (/usr/local/libexec/freshports) 
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: checking jobs for ingress (/usr/local/libexec/freshports) 
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: /var/db/ingress/signals/check_git exists.  About to run check_git.sh (/usr/local/libexec/freshports) 
Jun 27 19:37:34 devgit-ingress01 check_git.sh[74584]: /usr/local/libexec/freshports/check_git.sh has started

*** /var/log/freshports/git.log ***
2021.06.27 19:37:34 git-delta.sh has started. Will check these repos: 'doc ports src'
2021.06.27 19:37:34 git-delta.sh XML dir is /var/db/ingress/message-queues/incoming
2021.06.27 19:37:34 git-delta.sh Now processing repo: doc
2021.06.27 19:37:34 git-delta.sh REPODIR='/var/db/ingress/repos/doc' exists
2021.06.27 19:37:34 git-delta.sh Repodir is /var/db/ingress/repos/doc
2021.06.27 19:37:34 git-delta.sh Running: /usr/local/bin/git fetch:
2021.06.27 19:37:34 git-delta.sh Done.
looking at origin/HEAD
looking at origin/main
working on 'origin/main'
1a6d7511cc3a51bdddb132e3e66ee02d6edccb03
2021.06.27 19:37:34 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/main..origin/main
2021.06.27 19:37:34 git-delta.sh Done.
2021.06.27 19:37:34 git-delta.sh No commits were found
2021.06.27 19:37:34 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo doc --path /var/db/ingress/repos/doc ---commit-range /origin/main..origin/main --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: ${freshports/...}: Bad substitution
2021.06.27 19:37:34 git-delta.sh Now processing repo: ports
2021.06.27 19:37:34 git-delta.sh REPODIR='/var/db/ingress/repos/ports' exists
2021.06.27 19:37:34 git-delta.sh Repodir is /var/db/ingress/repos/ports
2021.06.27 19:37:34 git-delta.sh Running: /usr/local/bin/git fetch:
From https://git.FreeBSD.org/ports
   03ed448b5012..16bef8b16b7f  main       -> origin/main
   3a6240c21278..ae162bd98935  2021Q2     -> origin/2021Q2
2021.06.27 19:37:36 git-delta.sh Done.
looking at origin/2014Q1
looking at origin/2014Q2
looking at origin/2014Q3
looking at origin/2014Q4
looking at origin/2015Q1
looking at origin/2015Q2
looking at origin/2015Q3
looking at origin/2015Q4
looking at origin/2016Q1
looking at origin/2016Q2
looking at origin/2016Q3
looking at origin/2016Q4
looking at origin/2017Q1
looking at origin/2017Q2
looking at origin/2017Q3
looking at origin/2017Q4
looking at origin/2018Q1
looking at origin/2018Q2
looking at origin/2018Q3
looking at origin/2018Q4
looking at origin/2019Q1
looking at origin/2019Q2
looking at origin/2019Q3
looking at origin/2019Q4
looking at origin/2020Q1
looking at origin/2020Q2
looking at origin/2020Q3
looking at origin/2020Q4
looking at origin/2021Q1
looking at origin/2021Q2
working on 'origin/2021Q2'
'git tag -l freshports/origin/2021Q2' found nothing
Let's find the first commit in this branch
First ref is '4e3cf0163c4a00d4dac41d6da43472d2fcab2f29'
taging that now:
2021.06.27 19:37:36 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/2021Q2..origin/2021Q2
2021.06.27 19:37:36 git-delta.sh Done.
2021.06.27 19:37:36 git-delta.sh The commits found are:
2021.06.27 19:37:36 git-delta.sh ae162bd989359e2e599a2b9cb58da87bdec05fab
2021.06.27 19:37:36 git-delta.sh 3a6240c2127867111d6eb18e39f539e35e3ffad7

[redacted for brevity]

2021.06.27 19:37:37 git-delta.sh 144bb249fc6f49387b8e0ba2cd27ef5f61430224
2021.06.27 19:37:37 git-delta.sh 5fbb95157840f744df004acd2e9a56b68422a632
2021.06.27 19:37:37 git-delta.sh ba68dd511d88a2fb6c4fefb53bc401b0f86e3217
2021.06.27 19:37:37 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo ports --path /var/db/ingress/repos/ports ---commit-range /origin/2021Q2..origin/2021Q2 --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: ${freshports/...}: Bad substitution
2021.06.27 19:37:37 git-delta.sh Now processing repo: src
2021.06.27 19:37:37 git-delta.sh REPODIR='/var/db/ingress/repos/src' exists
2021.06.27 19:37:37 git-delta.sh Repodir is /var/db/ingress/repos/src
2021.06.27 19:37:37 git-delta.sh Running: /usr/local/bin/git fetch:
From https://git.FreeBSD.org/src
   b762974cf4b..c7f048ab353  main       -> origin/main
2021.06.27 19:37:38 git-delta.sh Done.
looking at origin/HEAD
looking at origin/main
working on 'origin/main'
cd8b62fb8c563b11dc6b7562b6a604258e73161b
2021.06.27 19:37:38 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/main..origin/main
2021.06.27 19:37:38 git-delta.sh Done.
2021.06.27 19:37:38 git-delta.sh The commits found are:
2021.06.27 19:37:38 git-delta.sh c7f048ab3532a9f081addd6da0adf96f25271de8
2021.06.27 19:37:38 git-delta.sh 914b6a0122c3032c2c2ae3a44bdeaf69e37efd61
2021.06.27 19:37:38 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo src --path /var/db/ingress/repos/src ---commit-range /origin/main..origin/main --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: ${freshports/...}: Bad substitution
2021.06.27 19:37:38 git-delta.sh Ending

*** /var/log/freshports/ingress-daemon.log ***
Jun 27 19:37:38 devgit-ingress01 ingress[74579]: job-waiting.pl finishes normally

*** /var/log/freshports/freshports.log ***
Jun 27 19:37:38 devgit-ingress01 check_git.sh[75322]: /usr/local/libexec/freshports/check_git.sh has finished
Jun 27 19:37:38 devgit-ingress01 FreshPorts[74582]: Finished running check_git.sh (/usr/local/libexec/freshports) 
Jun 27 19:37:38 devgit-ingress01 FreshPorts[74582]: flag '/var/db/ingress/signals/check_git' not set.  no work for check_git.sh (/usr/local/libexec/freshports) 

The main issue I see is the Bad substitution errors on line 47, 104, and 121. If you look back at lines 115-118 at the bottom of Processing commits on branches with git – part 2. In short:

${xfreshports}/${refname}..$refname

Became

freshports/${refname}..${refname}

I also see that I need to set the tag freshports/origin/2021Q2 on the ports repo (see line 87 above). I’ve already processed some of those commits last weekend. Looking at https://devgit.freshports.org/?branch=quarterly I see the last commit (as of when I type this) is 5ceea22.

This is how I set that tag:

[dan@devgit-ingress01:~] $ sudo su -l ingress
$ bash
[ingress@devgit-ingress01 ~]$ cd repos/ports
[ingress@devgit-ingress01 ~/repos/ports]$ git tag -m "last known commit of freshports/origin/2021Q2" -f freshports/origin/2021Q2 5ceea227c504d2892d91c1aa8d8d81ff15b22fc3
Updated tag 'freshports/origin/2021Q2' (was dbbd50cc2513)
[ingress@devgit-ingress01 ~/repos/ports]$ 

The second run

On the second run, the found commits looked good, but git-to-freshports-xml.py was bringing in port commits from 1994-08-21, doc commits from 1995-08-27, and src commits from 1993-06-12. In short, it was building XML files for the entire repo. I don’t want that, not right now. Someone day we can look at importing all commits, but I suspect there will be infrastructure changes which are not backwards compatible with port commits.

Now I’m into debugging and I won’t post all of this work.

I think /usr/local/libexec/freshports/git-to-freshports-xml.py needs to be updated.

Jun 272021
 

Following on from last weekend’s work, today I’m learning more about git, and the difference between Plumbing and Porcelain commands.

The original intent was to use git tag -l freshports/$refname to detect if a tag exists. The problem there, it always returns 0.

[dan@mydev:/var/db/ingress/repos/ports] $ git tag -l | head
10-eol
4-eol
5-eol
6-eol
7-eol
8-eol
9-eol
pkg-install-eol
pre-xorg-7
release/10.0.0
[dan@mydev:/var/db/ingress/repos/ports] $ 


[dan@mydev:/var/db/ingress/repos/ports] $ git tag -l 10-eol
10-eol
[dan@mydev:/var/db/ingress/repos/ports] $ echo $?
0
[dan@mydev:/var/db/ingress/repos/ports] $ git tag -l 10-eollkjadslkjdf
[dan@mydev:/var/db/ingress/repos/ports] $ echo $?
0
[dan@mydev:/var/db/ingress/repos/ports] $ git tag -l | grep 10-eollkjadslkjdf
[dan@mydev:/var/db/ingress/repos/ports] $ 

From the above, whether the tag exists or not, the code can’t tell, well, without looking at the output.

Let’s use this instead:

[dan@mydev:/var/db/ingress/repos/ports] $ git rev-parse --verify refs/tags/10-eol
1aa5d44ff272774be87b1afa6a71fa6cbd62c8af
[dan@mydev:/var/db/ingress/repos/ports] $ echo $?
0
[dan@mydev:/var/db/ingress/repos/ports] $ git rev-parse --verify refs/tags/10-eollkjadslkjdf
fatal: Needed a single revision
[dan@mydev:/var/db/ingress/repos/ports] $ echo $
[dan@mydev:/var/db/ingress/repos/ports] $ 

We can also use the -q to suppress output.

Next goal

Let’s try this on the devgit.freshports.org server next.

This is the current code:

#!/bin/sh

# process the new commits
# based upon https://github.com/FreshPorts/git_proc_commit/issues/3
# An idea from https://github.com/sarcasticadmin

if [ ! -f /usr/local/etc/freshports/config.sh ]
then
	echo "/usr/local/etc/freshports/config.sh.sh not found by $0"
	exit 1
fi

# this can be a space separated list of repositories to check
# e.g. "doc ports src"
repos=$1

. /usr/local/etc/freshports/config.sh

LOGGERTAG='git-delta.sh'

logfile "has started. Will check these repos: '${repos}'"

# what remote are we using on this repo?
REMOTE='origin'

# where we do dump the XML files which we create?
XML="${INGRESS_MSGDIR}/incoming"

logfile "XML dir is $XML"

for repo in ${repos}
do
   logfile "Now processing repo: ${repo}"

   # convert the repo label to a physical directory on disk
   dir=$(convert_repo_label_to_directory ${repo})

   # empty means error
   if [  "${dir}" == "" ]; then
      logfile "FATAL error, repo='${repo}' is unknown: cannot translate it to a directory name"
      continue
   fi

   # where is the repo directory?
   # This is the directory which contains the repos.
   REPODIR="${INGRESS_PORTS_DIR_BASE}/${dir}"
   LATEST_FILE="${INGRESS_PORTS_DIR_BASE}/latest.${dir}"

   if [ -d ${REPODIR} ]; then
      logfile "REPODIR='${REPODIR}' exists"
   else
      logfile "FATAL error, REPODIR='${REPODIR}' is not a directory"
      continue
   fi

   logfile "Repodir is $REPODIR"
   # on with the work

   cd ${REPODIR}

   # Bring local branch up-to-date with the local remote
   logfile "Running: ${GIT} fetch:"
   ${GIT} fetch
   logfile "Done."

   NAME_OF_HEAD="main"
   NAME_OF_REMOTE="origin"
   MAIN_BRANCH="$NAME_OF_REMOTE/$NAME_OF_HEAD"

   git for-each-ref --format '%(objecttype) %(refname)' \
      | sed -n 's/^commit refs\/remotes\///p' \
      | while read -r refname
   do
      echo looking at $refname
      # for now, when testing, only this branch please
      if [ "$refname" != "origin/2021Q2" ] && [ "$refname" != "$MAIN_BRANCH" ]
      then
         continue
      fi

      echo "working on '$refname'"

      if ! git rev-parse -q --verify freshports/$refname
      then
         if [ "$refname" == "$MAIN_BRANCH" ]
         then
            echo "FATAL - '$MAIN_BRANCH' must have tag 'freshports/$refname' set manually  - special case the main branch because the best merge base is the most recent commit"
            exit
         fi
         echo "'git tag -l freshports/$refname'" found nothing
         echo "Let's find the first commit in this branch"
         first_ref=$(git merge-base $NAME_OF_REMOTE/$NAME_OF_HEAD $refname)
         echo "First ref is '$first_ref'"
         # get the first commit of that branch and create a tag.
         echo taging that now:
         git tag -m "first known commit of $refname" -f freshports/$refname $first_ref
      fi

      # get list of commits, if only to document them here
      logfile "Running: ${GIT} rev-list freshports/$refname..$refname"
      commits=$(${GIT} rev-list freshports/$refname..$refname)
      logfile "Done."

      if [ -z "$commits" ]
      then
         logfile "No commits were found"
      else
        logfile "The commits found are:"
        for commit in $commits
        do
           logfile "$commit"
        done
      fi

      logfile "${SCRIPTDIR}/git-to-freshports-xml.py --repo ${repo} --path ${REPODIR} ---commit-range ${xfreshports}/${refname}..$refname --spooling ${INGRESS_SPOOLINGDIR} --output ${XML}"
      
      
               ${SCRIPTDIR}/git-to-freshports-xml.py --repo ${repo} --path ${REPODIR} ---commit-range ${freshports/$refname}..$refname --spooling ${INGRESS_SPOOLINGDIR} --output ${XML}

      new_latest=$(${GIT}  rev-parse HEAD)
      
      echo new_latest = $new_latest

      # echo $new_latest > ${LATEST_FILE}
      # Store the last known commit that we just processed.
      git tag -m "last known commit of $refname" -f freshports/$refname $refname

   done
done

logfile "Ending"