This outlines the steps for putting the new git-delta.sh script into use. These steps will be required on each ingress node.
stop ingress commit processing
Look at the output of sudo tail /var/log/freshports/git.log and you should see something like this:
[dan@devgit-ingress01:~] $ sudo tail /var/log/freshports/git.log 2021.06.27 17:21:05 git-delta.sh Repodir is /var/db/ingress/repos/src 2021.06.27 17:21:05 git-delta.sh Running: /usr/local/bin/git pull: Already up to date. 2021.06.27 17:21:06 git-delta.sh Done. 2021.06.27 17:21:06 git-delta.sh STARTPOINT = b762974cf4b9ea77f1decf4a6d829372f0a97f75 2021.06.27 17:21:06 git-delta.sh Running: /usr/local/bin/git rev-list b762974cf4b9ea77f1decf4a6d829372f0a97f75..HEAD 2021.06.27 17:21:06 git-delta.sh Done. 2021.06.27 17:21:06 git-delta.sh No commits were found 2021.06.27 17:21:06 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo src --path /var/db/ingress/repos/src --commit b762974cf4b9ea77f1decf4a6d829372f0a97f75 --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming 2021.06.27 17:21:06 git-delta.sh Ending
Since it was 13:21 and not close to a multiple of 3 (this script runs every 3 minutes), I knew that I could stop the ingress daemon:
[dan@devgit-ingress01:~] $ sudo service ingress stop Stopping ingress.
Add the tags
The existing configuration uses files to store the ‘last commit’. See here:
[dan@devgit-ingress01:/var/db/ingress/repos] $ ls -l total 28 drwxr-xr-x 7 ingress ingress 11 Jun 5 18:51 doc -rw-r--r-- 1 ingress ingress 41 Jun 27 17:21 latest.doc -rw-r--r-- 1 ingress ingress 41 Jun 27 17:21 latest.ports -rw-r--r-- 1 ingress ingress 41 Jun 27 17:21 latest.src drwxr-xr-x 70 ingress ingress 81 Jun 27 14:51 ports drwxr-xr-x 27 ingress ingress 44 Jun 27 01:00 src [dan@devgit-ingress01:/var/db/ingress/repos] $
This is the script I created for that:
#!/bin/sh # The main branch is always origin/main - so far refname='origin/main' repos="doc ports src" for repo in $repos do tag=$(cat latest.$repo) cd $repo git tag -m "last known commit of " -f freshports/origin/main $tag cd - done
Running that:
$ # The main branch is always origin/main - so far $ refname='origin/main' $ $ repos="doc ports src" $ for repo in $repos > do > tag=$(cat latest.$repo) > cd $repo > git tag -m "last known commit of " -f freshports/origin/main $tag > cd - > done /var/db/ingress/repos /var/db/ingress/repos /var/db/ingress/repos
Not entirely exciting.
This is the script to display the existing values:
#!/bin/sh
# The main branch is always origin/main - so far
refname='origin/main'
repos="doc ports src"
for repo in $repos
do
tag=$(cat latest.$repo)
cd $repo
git rev-parse --verify freshports/$refname^{}
cd -
done
Running that:
[dan@devgit-ingress01:/var/db/ingress/repos] $ /bin/sh $ # The main branch is always origin/main - so far $ refname='origin/main' $ $ repos="doc ports src" $ for repo in $repos > do > tag=$(cat latest.$repo) > cd $repo > git rev-parse --verify freshports/$refname > cd .. > done 048a537b787b9e858df140dfdc2122e3fb49c198 /var/db/ingress/repos cc60e220aa4fd4c3b0f5eb74b8eb13bcf6f9c406 /var/db/ingress/repos ca79c1fd81f2b2694a157df3beecb6fb48483900 /var/db/ingress/repos $ cat latest.doc 4778fc33a57b1a7b9597868fa7fadef63a891edd $ cat latest.ports 03ed448b501283f1e4d5dd0abb6847a91a32a79b $ cat latest.src b762974cf4b9ea77f1decf4a6d829372f0a97f75 $
That seems wrong. Why is the hash from the git rev-parse –verify not the same.
EDIT: 2021-07-09 The solution to seeing the actual commit hash is to change the line to:
git rev-parse --verify freshports/$refname^{}
That deferences the tag. The above code has since been updated with that change, for my convenience.
end EDIT.
I spent 30-45 minutes looking in that. Then I did this:
$ git show freshports/origin/main
tag freshports/origin/main
Tagger: Dan Langille <dan@langille.org>
Date: Sun Jun 27 18:59:35 2021 +0000
last known commit of origin/main
commit 03ed448b501283f1e4d5dd0abb6847a91a32a79b (tag: freshports/origin/main)
Author: Rene Ladan <rene@FreeBSD.org>
Date: Sun Jun 27 16:56:24 2021 +0200
sysutils/bhyve-firmware: undeprecate CSM option
The sysutils/uefi-edk2-bhyve port no longer needs Python 2.7
diff --git a/sysutils/bhyve-firmware/Makefile b/sysutils/bhyve-firmware/Makefile
index 1fb78f76049c..3c601a11f7ef 100644
--- a/sysutils/bhyve-firmware/Makefile
+++ b/sysutils/bhyve-firmware/Makefile
@@ -14,11 +14,4 @@ OPTIONS_DEFAULT= CSM
CSM_DESC= Include firmware with Compatibility Support Module
CSM_RUN_DEPENDS= uefi-edk2-bhyve-csm>=0:sysutils/uefi-edk2-bhyve-csm
-.include <bsd.port.options.mk>
That commit hash matches the value in latest.ports. All is good. This has done the right thing.
Move the script into place
I copied the script from mydev to test-ingress01:
$ scp git-delta.sh devgit-ingress01: git-delta.sh
Then moved it into to place:
[dan@devgit-ingress01:/var/db/ingress/repos] $ mv -i ~/git-delta.sh /usr/local/libexec/freshports/git-delta.sh [dan@devgit-ingress01:/var/db/ingress/repos] $ sudo pkg check -sr Checking all packages: 87% py38-freshports-git-proc-commit-0.0.4_1: checksum mismatch for /usr/local/libexec/freshports/git-delta.sh Checking all packages: 100% [dan@devgit-ingress01:/var/db/ingress/repos] $ ls -l /usr/local/libexec/freshports/git-delta.sh -rwxr-xr-x 1 dan dan 3977 Jun 27 18:57 /usr/local/libexec/freshports/git-delta.sh
Yep, that’s the right file.
I’m overwriting the file usually installed by package. Most of /usr/local/libexec/freshports/ is part of an svn co of the code. But a few files are not.
[dan@devgit-ingress01:/usr/local/libexec/freshports] $ svn st ? .idea ? UpdatePackagesFromRawPackages.py ? check_freshports_cache_clearing ? check_git.sh ? check_report_logs.pl ? check_repos_for_new_stuff.py ? fetch-extract-parse-import-one-abi.sh ? get_packagesite.txz_date ? git-delta.sh ? git-range-of-commits.sh ? git-single-commit.sh ? git-to-freshports-xml.py ? helper_scripts ? import-via-copy-packagesite-all-raw-fields.py ? import_packagesite.py ? show-config.sh ? test-categories.pl ? test-new-xml-code-single.sh ? test-new-xml-code-starting-from.sh ? testing.pm [dan@devgit-ingress01:/usr/local/libexec/freshports] $ pkg info -l py38-freshports-git-proc-commit py38-freshports-git-proc-commit-0.0.4_1: /usr/local/libexec/freshports/check_git.sh /usr/local/libexec/freshports/git-delta.sh /usr/local/libexec/freshports/git-range-of-commits.sh /usr/local/libexec/freshports/git-single-commit.sh /usr/local/libexec/freshports/git-to-freshports-xml.py [dan@devgit-ingress01:/usr/local/libexec/freshports] $
Now let’s wake up the sleeping ingress daemon.
Trying out the script
I’m going to run the ingress daemon after stopping the freshports daemon. I want to look for incoming commits and create the XML, but I don’t want to process the XML.
$ sudo service freshports stop Stopping freshports.
Now start:
[dan@devgit-ingress01:~] $ sudo service ingress start Starting ingress.
And the logs pile up:
[dan@devgit-ingress01:/usr/local/libexec/freshports] $ sudo xtail /var/log/freshports/*.log
^C
*** recently changed files ***
1 27-Jun-21 19:36:09 /var/log/freshports/ingress-daemon.log
2 27-Jun-21 19:36:09 /var/log/freshports/freshports.log
3 27-Jun-21 17:21:06 /var/log/freshports/git.log
4 27-Jun-21 17:15:11 /var/log/freshports/freshports-daemon.log
5 27-Jun-21 03:04:23 /var/log/freshports/report-new-ports.daily.log
6 27-Jun-21 03:04:22 /var/log/freshports/report-notification.daily.log
7 27-Jun-21 03:03:01 /var/log/freshports/report-security.daily.log
8 27-Jun-21 00:00:00 /var/log/freshports/report-notification.weekly.log
currently watching: 15 files 0 dirs 0 unknown entries
*** /var/log/freshports/ingress-daemon.log ***
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: starting up!
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: incoming: /var/db/ingress/message-queues/incoming
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: ready
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: yes, there is a job waiting
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: running /usr/local/bin/perl ./job-waiting.pl
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: from directory /usr/local/libexec/freshports
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: -rw-r--r-- 1 dan dan 2421 May 9 12:36 ./job-waiting.pl
*** /var/log/freshports/freshports.log ***
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: running job-waiting.pl
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: starting ./job-waiting.pl (/usr/local/libexec/freshports)
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: running ./job-waiting.pl as user = 'ingress' (/usr/local/libexec/freshports)
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: checking jobs for ingress (/usr/local/libexec/freshports)
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: /var/db/ingress/signals/check_git exists. About to run check_git.sh (/usr/local/libexec/freshports)
Jun 27 19:37:34 devgit-ingress01 check_git.sh[74584]: /usr/local/libexec/freshports/check_git.sh has started
*** /var/log/freshports/git.log ***
2021.06.27 19:37:34 git-delta.sh has started. Will check these repos: 'doc ports src'
2021.06.27 19:37:34 git-delta.sh XML dir is /var/db/ingress/message-queues/incoming
2021.06.27 19:37:34 git-delta.sh Now processing repo: doc
2021.06.27 19:37:34 git-delta.sh REPODIR='/var/db/ingress/repos/doc' exists
2021.06.27 19:37:34 git-delta.sh Repodir is /var/db/ingress/repos/doc
2021.06.27 19:37:34 git-delta.sh Running: /usr/local/bin/git fetch:
2021.06.27 19:37:34 git-delta.sh Done.
looking at origin/HEAD
looking at origin/main
working on 'origin/main'
1a6d7511cc3a51bdddb132e3e66ee02d6edccb03
2021.06.27 19:37:34 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/main..origin/main
2021.06.27 19:37:34 git-delta.sh Done.
2021.06.27 19:37:34 git-delta.sh No commits were found
2021.06.27 19:37:34 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo doc --path /var/db/ingress/repos/doc ---commit-range /origin/main..origin/main --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: ${freshports/...}: Bad substitution
2021.06.27 19:37:34 git-delta.sh Now processing repo: ports
2021.06.27 19:37:34 git-delta.sh REPODIR='/var/db/ingress/repos/ports' exists
2021.06.27 19:37:34 git-delta.sh Repodir is /var/db/ingress/repos/ports
2021.06.27 19:37:34 git-delta.sh Running: /usr/local/bin/git fetch:
From https://git.FreeBSD.org/ports
03ed448b5012..16bef8b16b7f main -> origin/main
3a6240c21278..ae162bd98935 2021Q2 -> origin/2021Q2
2021.06.27 19:37:36 git-delta.sh Done.
looking at origin/2014Q1
looking at origin/2014Q2
looking at origin/2014Q3
looking at origin/2014Q4
looking at origin/2015Q1
looking at origin/2015Q2
looking at origin/2015Q3
looking at origin/2015Q4
looking at origin/2016Q1
looking at origin/2016Q2
looking at origin/2016Q3
looking at origin/2016Q4
looking at origin/2017Q1
looking at origin/2017Q2
looking at origin/2017Q3
looking at origin/2017Q4
looking at origin/2018Q1
looking at origin/2018Q2
looking at origin/2018Q3
looking at origin/2018Q4
looking at origin/2019Q1
looking at origin/2019Q2
looking at origin/2019Q3
looking at origin/2019Q4
looking at origin/2020Q1
looking at origin/2020Q2
looking at origin/2020Q3
looking at origin/2020Q4
looking at origin/2021Q1
looking at origin/2021Q2
working on 'origin/2021Q2'
'git tag -l freshports/origin/2021Q2' found nothing
Let's find the first commit in this branch
First ref is '4e3cf0163c4a00d4dac41d6da43472d2fcab2f29'
taging that now:
2021.06.27 19:37:36 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/2021Q2..origin/2021Q2
2021.06.27 19:37:36 git-delta.sh Done.
2021.06.27 19:37:36 git-delta.sh The commits found are:
2021.06.27 19:37:36 git-delta.sh ae162bd989359e2e599a2b9cb58da87bdec05fab
2021.06.27 19:37:36 git-delta.sh 3a6240c2127867111d6eb18e39f539e35e3ffad7
[redacted for brevity]
2021.06.27 19:37:37 git-delta.sh 144bb249fc6f49387b8e0ba2cd27ef5f61430224
2021.06.27 19:37:37 git-delta.sh 5fbb95157840f744df004acd2e9a56b68422a632
2021.06.27 19:37:37 git-delta.sh ba68dd511d88a2fb6c4fefb53bc401b0f86e3217
2021.06.27 19:37:37 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo ports --path /var/db/ingress/repos/ports ---commit-range /origin/2021Q2..origin/2021Q2 --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: ${freshports/...}: Bad substitution
2021.06.27 19:37:37 git-delta.sh Now processing repo: src
2021.06.27 19:37:37 git-delta.sh REPODIR='/var/db/ingress/repos/src' exists
2021.06.27 19:37:37 git-delta.sh Repodir is /var/db/ingress/repos/src
2021.06.27 19:37:37 git-delta.sh Running: /usr/local/bin/git fetch:
From https://git.FreeBSD.org/src
b762974cf4b..c7f048ab353 main -> origin/main
2021.06.27 19:37:38 git-delta.sh Done.
looking at origin/HEAD
looking at origin/main
working on 'origin/main'
cd8b62fb8c563b11dc6b7562b6a604258e73161b
2021.06.27 19:37:38 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/main..origin/main
2021.06.27 19:37:38 git-delta.sh Done.
2021.06.27 19:37:38 git-delta.sh The commits found are:
2021.06.27 19:37:38 git-delta.sh c7f048ab3532a9f081addd6da0adf96f25271de8
2021.06.27 19:37:38 git-delta.sh 914b6a0122c3032c2c2ae3a44bdeaf69e37efd61
2021.06.27 19:37:38 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo src --path /var/db/ingress/repos/src ---commit-range /origin/main..origin/main --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: ${freshports/...}: Bad substitution
2021.06.27 19:37:38 git-delta.sh Ending
*** /var/log/freshports/ingress-daemon.log ***
Jun 27 19:37:38 devgit-ingress01 ingress[74579]: job-waiting.pl finishes normally
*** /var/log/freshports/freshports.log ***
Jun 27 19:37:38 devgit-ingress01 check_git.sh[75322]: /usr/local/libexec/freshports/check_git.sh has finished
Jun 27 19:37:38 devgit-ingress01 FreshPorts[74582]: Finished running check_git.sh (/usr/local/libexec/freshports)
Jun 27 19:37:38 devgit-ingress01 FreshPorts[74582]: flag '/var/db/ingress/signals/check_git' not set. no work for check_git.sh (/usr/local/libexec/freshports)
The main issue I see is the Bad substitution errors on line 47, 104, and 121. If you look back at lines 115-118 at the bottom of Processing commits on branches with git – part 2. In short:
${xfreshports}/${refname}..$refname
Became
freshports/${refname}..${refname}
I also see that I need to set the tag freshports/origin/2021Q2 on the ports repo (see line 87 above). I’ve already processed some of those commits last weekend. Looking at https://devgit.freshports.org/?branch=quarterly I see the last commit (as of when I type this) is 5ceea22.
This is how I set that tag:
[dan@devgit-ingress01:~] $ sudo su -l ingress $ bash [ingress@devgit-ingress01 ~]$ cd repos/ports [ingress@devgit-ingress01 ~/repos/ports]$ git tag -m "last known commit of freshports/origin/2021Q2" -f freshports/origin/2021Q2 5ceea227c504d2892d91c1aa8d8d81ff15b22fc3 Updated tag 'freshports/origin/2021Q2' (was dbbd50cc2513) [ingress@devgit-ingress01 ~/repos/ports]$
The second run
On the second run, the found commits looked good, but git-to-freshports-xml.py was bringing in port commits from 1994-08-21, doc commits from 1995-08-27, and src commits from 1993-06-12. In short, it was building XML files for the entire repo. I don’t want that, not right now. Someone day we can look at importing all commits, but I suspect there will be infrastructure changes which are not backwards compatible with port commits.
Now I’m into debugging and I won’t post all of this work.
I think /usr/local/libexec/freshports/git-to-freshports-xml.py needs to be updated.
