Putting the new git-delta.sh into use on devgit.freshports.org

This outlines the steps for putting the new git-delta.sh script into use. These steps will be required on each ingress node.

stop ingress commit processing

Look at the output of sudo tail /var/log/freshports/git.log and you should see something like this:

[dan@devgit-ingress01:~] $ sudo tail /var/log/freshports/git.log
2021.06.27 17:21:05 git-delta.sh Repodir is /var/db/ingress/repos/src
2021.06.27 17:21:05 git-delta.sh Running: /usr/local/bin/git pull:
Already up to date.
2021.06.27 17:21:06 git-delta.sh Done.
2021.06.27 17:21:06 git-delta.sh STARTPOINT = b762974cf4b9ea77f1decf4a6d829372f0a97f75
2021.06.27 17:21:06 git-delta.sh Running: /usr/local/bin/git rev-list b762974cf4b9ea77f1decf4a6d829372f0a97f75..HEAD
2021.06.27 17:21:06 git-delta.sh Done.
2021.06.27 17:21:06 git-delta.sh No commits were found
2021.06.27 17:21:06 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo src --path /var/db/ingress/repos/src --commit b762974cf4b9ea77f1decf4a6d829372f0a97f75 --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
2021.06.27 17:21:06 git-delta.sh Ending

Since it was 13:21 and not close to a multiple of 3 (this script runs every 3 minutes), I knew that I could stop the ingress daemon:

[dan@devgit-ingress01:~] $ sudo service ingress stop
Stopping ingress.

Add the tags

The existing configuration uses files to store the ‘last commit’. See here:

[dan@devgit-ingress01:/var/db/ingress/repos] $ ls -l
total 28
drwxr-xr-x   7 ingress  ingress  11 Jun  5 18:51 doc
-rw-r--r--   1 ingress  ingress  41 Jun 27 17:21 latest.doc
-rw-r--r--   1 ingress  ingress  41 Jun 27 17:21 latest.ports
-rw-r--r--   1 ingress  ingress  41 Jun 27 17:21 latest.src
drwxr-xr-x  70 ingress  ingress  81 Jun 27 14:51 ports
drwxr-xr-x  27 ingress  ingress  44 Jun 27 01:00 src
[dan@devgit-ingress01:/var/db/ingress/repos] $ 

This is the script I created for that:

Running that:

$ # The main branch is always origin/main - so far
$ refname='origin/main'
$ 
$ repos="doc ports src"
$ for repo in $repos
> do
>   tag=$(cat latest.$repo)
>   cd $repo
>   git tag -m "last known commit of " -f freshports/origin/main $tag
>   cd -
> done
/var/db/ingress/repos
/var/db/ingress/repos
/var/db/ingress/repos

Not entirely exciting.

This is the script to display the existing values:

Running that:

[dan@devgit-ingress01:/var/db/ingress/repos] $ /bin/sh
$ # The main branch is always origin/main - so far
$ refname='origin/main'
$ 
$ repos="doc ports src"
$ for repo in $repos
> do
>   tag=$(cat latest.$repo)
>   cd $repo
>   git rev-parse --verify freshports/$refname
>   cd ..
> done
048a537b787b9e858df140dfdc2122e3fb49c198
/var/db/ingress/repos
cc60e220aa4fd4c3b0f5eb74b8eb13bcf6f9c406
/var/db/ingress/repos
ca79c1fd81f2b2694a157df3beecb6fb48483900
/var/db/ingress/repos
$ cat latest.doc
4778fc33a57b1a7b9597868fa7fadef63a891edd
$ cat latest.ports
03ed448b501283f1e4d5dd0abb6847a91a32a79b
$ cat latest.src
b762974cf4b9ea77f1decf4a6d829372f0a97f75
$ 

That seems wrong. Why is the hash from the git rev-parse –verify not the same.

---

EDIT: 2021-07-09 The solution to seeing the actual commit hash is to change the line to:

That deferences the tag. The above code has since been updated with that change, for my convenience.

end EDIT.

---

I spent 30-45 minutes looking in that. Then I did this:

That commit hash matches the value in latest.ports. All is good. This has done the right thing.

Move the script into place

I copied the script from mydev to test-ingress01:

$ scp git-delta.sh devgit-ingress01:
git-delta.sh                                                                                                                                                              

Then moved it into to place:

[dan@devgit-ingress01:/var/db/ingress/repos] $ mv -i ~/git-delta.sh  /usr/local/libexec/freshports/git-delta.sh
[dan@devgit-ingress01:/var/db/ingress/repos] $ sudo pkg check -sr
Checking all packages:  87%
py38-freshports-git-proc-commit-0.0.4_1: checksum mismatch for /usr/local/libexec/freshports/git-delta.sh
Checking all packages: 100%
[dan@devgit-ingress01:/var/db/ingress/repos] $ ls -l /usr/local/libexec/freshports/git-delta.sh
-rwxr-xr-x  1 dan  dan  3977 Jun 27 18:57 /usr/local/libexec/freshports/git-delta.sh

Yep, that’s the right file.

I’m overwriting the file usually installed by package. Most of /usr/local/libexec/freshports/ is part of an svn co of the code. But a few files are not.

[dan@devgit-ingress01:/usr/local/libexec/freshports] $ svn st
?       .idea
?       UpdatePackagesFromRawPackages.py
?       check_freshports_cache_clearing
?       check_git.sh
?       check_report_logs.pl
?       check_repos_for_new_stuff.py
?       fetch-extract-parse-import-one-abi.sh
?       get_packagesite.txz_date
?       git-delta.sh
?       git-range-of-commits.sh
?       git-single-commit.sh
?       git-to-freshports-xml.py
?       helper_scripts
?       import-via-copy-packagesite-all-raw-fields.py
?       import_packagesite.py
?       show-config.sh
?       test-categories.pl
?       test-new-xml-code-single.sh
?       test-new-xml-code-starting-from.sh
?       testing.pm
[dan@devgit-ingress01:/usr/local/libexec/freshports] $ pkg info -l py38-freshports-git-proc-commit 
py38-freshports-git-proc-commit-0.0.4_1:
	/usr/local/libexec/freshports/check_git.sh
	/usr/local/libexec/freshports/git-delta.sh
	/usr/local/libexec/freshports/git-range-of-commits.sh
	/usr/local/libexec/freshports/git-single-commit.sh
	/usr/local/libexec/freshports/git-to-freshports-xml.py
[dan@devgit-ingress01:/usr/local/libexec/freshports] $ 

Now let’s wake up the sleeping ingress daemon.

Trying out the script

I’m going to run the ingress daemon after stopping the freshports daemon. I want to look for incoming commits and create the XML, but I don’t want to process the XML.

$ sudo service freshports stop
Stopping freshports.

Now start:

[dan@devgit-ingress01:~] $ sudo service ingress start
Starting ingress.

And the logs pile up:

The main issue I see is the Bad substitution errors on line 47, 104, and 121. If you look back at lines 115-118 at the bottom of Processing commits on branches with git – part 2. In short:

${xfreshports}/${refname}..$refname

Became

freshports/${refname}..${refname}

I also see that I need to set the tag freshports/origin/2021Q2 on the ports repo (see line 87 above). I’ve already processed some of those commits last weekend. Looking at https://devgit.freshports.org/?branch=quarterly I see the last commit (as of when I type this) is 5ceea22.

This is how I set that tag:

[dan@devgit-ingress01:~] $ sudo su -l ingress
$ bash
[ingress@devgit-ingress01 ~]$ cd repos/ports
[ingress@devgit-ingress01 ~/repos/ports]$ git tag -m "last known commit of freshports/origin/2021Q2" -f freshports/origin/2021Q2 5ceea227c504d2892d91c1aa8d8d81ff15b22fc3
Updated tag 'freshports/origin/2021Q2' (was dbbd50cc2513)
[ingress@devgit-ingress01 ~/repos/ports]$ 

The second run

On the second run, the found commits looked good, but git-to-freshports-xml.py was bringing in port commits from 1994-08-21, doc commits from 1995-08-27, and src commits from 1993-06-12. In short, it was building XML files for the entire repo. I don’t want that, not right now. Someone day we can look at importing all commits, but I suspect there will be infrastructure changes which are not backwards compatible with port commits.

Now I’m into debugging and I won’t post all of this work.

I think /usr/local/libexec/freshports/git-to-freshports-xml.py needs to be updated.