Dan Langille

I've been playing with Open Source software, starting with FreeBSD, since New Zealand Post installed DSL on my street in 1998. From there, I started writing at The FreeBSD Diary, moving my work here after I discovered WordPress. Along the way, I started the BSDCan and PGCon conferences. I slowly moved from software development into full time systems administration and now work for very-well known company who has been a big force in the security industry.

Jul 042021
 

Right now, we have have:

  • dev.freshports.org
  • devgit.freshports.org
  • test.freshports.org
  • testgit.freshports.org
  • stage.freshports.org
  • stagegit.freshports.org

The goal: three hosts: dev, test, stage.

I’ll delete the existing: dev, test, stage

I’ll rename:

  • devgit -> dev
  • testgit -> test
  • stagegit -> stage

I think this will be easier than moving content from one jail to another.

Each of the three renamed hosts will have redirects to the new hosts.

EDIT: 2021-07-05 This work has been completed.

Jul 042021
 

Ports come and go. Sometimes they come back. Take a recent commit on 1 July against editors/ved (and others). This port was created on 01 Jan 2001 and removed on 15 May 2014 because it used the deprecated smake. The 1 July 2021 commit brought it back. There was a problem.

Here is the page after the commit:

FreshPorts page for editors/ved

FreshPorts page for editors/ved

The history shows that the version did not change with this commit.

FreshPorts history for editors/ved showing 1.71_1 for most recent commit and the previous commit

FreshPorts history for editors/ved showing 1.71_1 for most recent commit and the previous commit

If you go back into the 11 Apr 2014 commit for editors/ved and click on the Subversion link, you’ll wind up at revision 350895, you can see the previous Makefile.

With that 1 July commit, editors/ved became a secondary port and obtains its PORTVERSION information from the primary port devel/schilybase.

Looking at devel/schilybase, its version is 2021.06.07. The maintainer brought this issue to my attention.

Logs

In the logs I found:

[dan@devgit-ingress01:~/modules] $ cd ~freshports/message-queues/archive/2021_07/2021_07_01/
[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ less *2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3*.log
...
# # # # Resurrecting deleted ports # # # #
...
found a deleted port: port='ved', port_id='4397', element_id='48745'
...
  inspecting: 'Add' , '/ports/head/editors/ved/Makefile' , 'editors/ved/Makefile' , '2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3' , 'editors' , 'ved' , 'Makefile'
  ...found a file from that port
  ...hmmm, we are modifying a file for a port which is deleted...  ........ I will resurrect that port for you
Port editors/ved needs to be Resurrected
  ........ resurrection done!
  we will not examine any more files for this port!
  finished looking through the files
# # # # Finished resurrecting deleted ports # # # #

That last line is line 5937 of the log file.

That’s good. The code correctly detected that this commit was touching a dead port and brought it back to life. In the background, this was flipping a flag in the database from INACTIVE to ACTIVE.

But wait, there’s more, around line 7558 of this 14994 line file:

...
port = editors/ved, port_id = '4397', category_id='1', needs_refresh='7'
This port is deleted: not refreshing.
This port is deleted: not saving.
...

The code thinks editors/ved is deleted, and therefore does not refresh the database with the values it could extract from the Makefile.

What?

This is a left-hand right-hand situation where the code is reading a list of ports from the database at the start of the processing, modifying the database (setting the deleted port to be undeleted), and then not updating its local copy of that port.

It can be successfully argued that the code which does the deletion should be the setting that flag in the local copy. I think that is the way to go. However, for now, just for testing, I’m going to try this:

[dan@devgit-ingress01:~/modules] $ svn di
Index: verifyport.pm
===================================================================
--- verifyport.pm	(revision 5736)
+++ verifyport.pm	(working copy)
@@ -972,6 +972,8 @@
 				$extra         = $extra         // '';
 				print "  inspecting: '$action' , '$filename' , '$filename_stripped' , '$revision' , '$category_name' , '$port_name' , '$extra'\n";
 
+				# instead of doing this through $element, perhaps do it through $port
+				# when then invokes $element and then sets $port->{status}.
 				if ($category_name eq $port->{category} && $port->{name} eq $port_name) {
 					print "  ...found a file from that port\n";
 					if ($action eq $FreshPorts::Constants::ADD || $action eq $FreshPorts::Constants::MODIFY) {
@@ -983,6 +985,9 @@
 						$element->update_status();
 						print "  ........ resurrection done!\n";
 						print "  we will not examine any more files for this port!\n";
+
+						# For later, when we are updating ports, mark this port as active.
+						$port->{status} = $FreshPorts::Element::Active;
 						# we are looping through files for a single port, only resurrect once.
 						last;
 					}
[dan@devgit-ingress01:~/modules] $ 

Preparing the database for a test run

I’ll run this through the devgit database first. Grabbing the element id from the logs above, I ran this query:

freshports.devgit=# select *, element_pathname(id) from element where id = 48745;
  id   | name | parent_id | directory_file_flag | status |    element_pathname     
-------+------+-----------+---------------------+--------+-------------------------
 48745 | ved  |         2 | D                   | A      | /ports/head/editors/ved
(1 row)

freshports.devgit=# 

In order to ‘delete’, this port, I can run this query:

freshports.devgit=# begin;
BEGIN
freshports.devgit=# update element set status = 'D' where id = 48745;
UPDATE 1
freshports.devgit=# select *, element_pathname(id) from element where id = 48745;
  id   | name | parent_id | directory_file_flag | status |    element_pathname     
-------+------+-----------+---------------------+--------+-------------------------
 48745 | ved  |         2 | D                   | D      | /ports/head/editors/ved
(1 row)

freshports.devgit=# commit;
COMMIT
freshports.devgit=# 

Now when I view this port on devgit, I see:

editors/ved, now deleted (see the RIP tombstone)

editors/ved, now deleted (see the RIP tombstone)

The test run

This is how I rerun the commit. First, it is deleted from the database:

freshports.devgit=# begin;
BEGIN
freshports.devgit=# delete from commit_log where message_id = '2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3';
DELETE 1
freshports.devgit=# commit;
COMMIT
freshports.devgit=# 

Then, it is injected into the processing stream:

[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ ls *2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3.xml
[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ sudo mv -i 2021.07.01.21.11.52.000000.2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3.xml ~ingress/message-queues/incoming/
[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ 

A few seconds later, we have these files:

[dan@devgit-ingress01:/var/db/freshports/message-queues/archive/2021_07/2021_07_01] $ cd ~freshports/message-queues/recent/
[dan@devgit-ingress01:/var/db/freshports/message-queues/recent] $ ls -l *2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3*
-rw-r--r--  1 freshports  freshports  891981 Jul  4 18:06 2021.07.01.21.11.52.000000.2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3.log
-rw-rw-r--  1 freshports  freshports   11993 Jul  1 21:15 2021.07.01.21.11.52.000000.2a40da7011b3545ea4d9c3ec48eed369cc3b1aa3.xml
[dan@devgit-ingress01:/var/db/freshports/message-queues/recent] $ 

The editors/ved port now looks like this:

editors/ved with the correct version: 2021.06.07

editors/ved with the correct version: 2021.06.07

That’s editors/ved with the correct version: 2021.06.07

Success!

The most time consuming part of this exercise was tracing through the code to ensure that the values used by one part of the code matched the other part (left-hand vs right-hand mentioned above).

The logs again

This time the log was about 15400 lines and contained this at about line 8527:

port = editors/ved, port_id = '4397', category_id='1', needs_refresh='7'
into RefreshFromFiles()
working with repo='ports'
working on CommitBranch='main'
with svn_revision=''
this port *** EITHER *** does not need a refresh *** OR *** we were told not to fetch
RepoType='git'
Repository='ports'
CommitBranch='main'
calling FreshPorts::Branches::GetPathToRepoForBranch
CommitBranch:   'main
REPODIR:        '/var/db/freshports/ports-jail/var/db/repos/ports'
REPODIR_CHROOT: '/var/db/repos/ports'
checking to make sure /var/db/freshports/ports-jail/var/db/repos/ports/editors/ved/Makefile exists
Phew.  It's here.  Moving on....
makecommand = /usr/local/bin/sudo /usr/sbin/chroot -u freshports /var/db/freshports/ports-jail /make-port.sh /var/db/repos/ports editors/ved 2>/tmp/FreshPorts.editors.ved.make-error.2021.7.4.18.6.22.59948
Result = 0
trying to get master sites.  Errors will be in '/tmp/FreshPorts.editors.ved.make-mastersites-error.2021.7.4.18.6.23.59948'
'/usr/local/bin/sudo /usr/sbin/chroot -u freshports /var/db/freshports/ports-jail /make-master-sites-all.sh /var/db/repos/ports editors/ved 2>/tmp/FreshPorts.editors.ved.make-mastersites-error.2021.7.4.18
trying to get showconfig.  Errors will be in '/tmp/FreshPorts.editors.ved.showconfig.2021.7.4.18.6.23.59948'
'/usr/local/bin/sudo /usr/sbin/chroot -u freshports /var/db/freshports/ports-jail /make-showconfig.sh /var/db/repos/ports editors/ved 2>/tmp/FreshPorts.editors.ved.showconfig.2021.7.4.18.6.23.59948'
$result='0'
$showconfig='===> The following configuration options are available for ved-2021.06.07:
     DOCS=on: Build and/or install documentation
===> Use 'make config' to modify these settings'
 portname                 = 'ved'
 packagename              = 'ved'
...

This time the logs show that the port was being refreshed and the database updated.

Updating the code

While reviewing the ports code, I saw:

[dan@devgit-ingress01:~/modules] $ grep sub port.pm
sub _initialize {
sub _GetValuesFromRow {
sub new {
sub save {
sub savePortTableOnly {
sub _save {
sub FetchByID {
sub FetchByPartialPathName {
sub _FetchElementIDByPartialPathName {
sub _ExtractValuesFromMakefile {
sub _Validate {
sub _GetDescrAndHomePage($) {
sub _GetFileContentsFromJail($) {
sub _GetRealPath($) {
sub RefreshFromFiles($;$;$;$;$;$) {
sub GetNeedsRefreshForNewPort {
sub IsActive {
sub IsDeleted {
sub SetActive {
sub SetDeleted {
sub IsValidDate($) {
sub upate_generate_plist {
sub update_package_flavors {
sub update_depends {
sub depends_type_long {
sub update_depends_helper {
sub _addMissingPORTSDIR {
sub CreatePortOnBranch {
[dan@devgit-ingress01:~/modules] $ 

See the SetDeleted function there on line 18? That’s much better than what I initially had in my test code.

Instead of this:

+						# For later, when we are updating ports, mark this port as active.
+						$port->{status} = $FreshPorts::Element::Active;

I now have this:

+						# For later, when we are updating ports, mark this port as active.
+						$port->SetActive();

It works and uses the proper function too. However, it does not undelete the port. It just sets a value in the local copy.

More work is required there. I think a $port->Undelete() function is called for.

The patch

This is the tested (on exactly one commit) patch:

[dan@devgit-ingress01:~/modules] $ svn di
Index: port.pm
===================================================================
--- port.pm	(revision 5713)
+++ port.pm	(working copy)
@@ -199,7 +199,7 @@
 	my $CommitBranch = shift;
 	my $FullSave     = shift;
 
-	print "into FreshPorts::Port::save\n";
+	print "into FreshPorts::Port::_save\n";
 
 	#
 	# to save, element_id and category_id must be valid
@@ -352,6 +352,26 @@
 	return $this->{id};
 }
 
+sub Undelete {
+	my $this = shift;
+	my $dbh  = shift;
+
+	print "into FreshPorts::Port::Undelete\n";
+	my $element  = FreshPorts::Element->new($dbh);
+
+	$element->{id}     = $this->{element_id};
+	$element->{status} = $FreshPorts::Element::Active;
+
+	$element->update_status();
+
+	# For later, when we are updating ports, mark this port as active.
+	$this->SetActive();
+
+	print "leaving FreshPorts::Port::Undelete\n";
+
+	return $this->{id};
+}
+
 sub FetchByID {
 	my $this	= shift;
 
Index: verifyport.pm
===================================================================
--- verifyport.pm	(revision 5736)
+++ verifyport.pm	(working copy)
@@ -952,7 +952,7 @@
 	#
 	print "# # # # Resurrecting deleted ports # # # #\n\n";
 	while (my ($portname, $port) = each %Ports) {
-		if ($port->{status} eq $FreshPorts::Element::Deleted) {
+		if ($port->IsDeleted()) {
 			print "found a deleted port: port='$port->{name}', port_id='$port->{id}', element_id='$port->{element_id}'\n";
 			print "now looking for files which were modified...\n";
 
@@ -972,6 +972,8 @@
 				$extra         = $extra         // '';
 				print "  inspecting: '$action' , '$filename' , '$filename_stripped' , '$revision' , '$category_name' , '$port_name' , '$extra'\n";
 
+				# instead of doing this through $element, perhaps do it through $port
+				# when then invokes $element and then sets $port->{status}.
 				if ($category_name eq $port->{category} && $port->{name} eq $port_name) {
 					print "  ...found a file from that port\n";
 					if ($action eq $FreshPorts::Constants::ADD || $action eq $FreshPorts::Constants::MODIFY) {
@@ -978,11 +980,11 @@
 						print "  ...hmmm, we are modifying a file for a port which is deleted...";
 						print "  ........ I will resurrect that port for you\n";
 						FreshPorts::Utilities::ReportError('notice', "Port $category_name/$port_name needs to be Resurrected", 0);
-						$element->{id}     = $port->{element_id};
-						$element->{status} = $FreshPorts::Element::Active;
-						$element->update_status();
+						$port->Undelete($dbh);
+
 						print "  ........ resurrection done!\n";
 						print "  we will not examine any more files for this port!\n";
+
 						# we are looping through files for a single port, only resurrect once.
 						last;
 					}
[dan@devgit-ingress01:~/modules] $ 
Jun 272021
 

This outlines the steps for putting the new git-delta.sh script into use. These steps will be required on each ingress node.

stop ingress commit processing

Look at the output of sudo tail /var/log/freshports/git.log and you should see something like this:

[dan@devgit-ingress01:~] $ sudo tail /var/log/freshports/git.log
2021.06.27 17:21:05 git-delta.sh Repodir is /var/db/ingress/repos/src
2021.06.27 17:21:05 git-delta.sh Running: /usr/local/bin/git pull:
Already up to date.
2021.06.27 17:21:06 git-delta.sh Done.
2021.06.27 17:21:06 git-delta.sh STARTPOINT = b762974cf4b9ea77f1decf4a6d829372f0a97f75
2021.06.27 17:21:06 git-delta.sh Running: /usr/local/bin/git rev-list b762974cf4b9ea77f1decf4a6d829372f0a97f75..HEAD
2021.06.27 17:21:06 git-delta.sh Done.
2021.06.27 17:21:06 git-delta.sh No commits were found
2021.06.27 17:21:06 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo src --path /var/db/ingress/repos/src --commit b762974cf4b9ea77f1decf4a6d829372f0a97f75 --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
2021.06.27 17:21:06 git-delta.sh Ending

Since it was 13:21 and not close to a multiple of 3 (this script runs every 3 minutes), I knew that I could stop the ingress daemon:

[dan@devgit-ingress01:~] $ sudo service ingress stop
Stopping ingress.

Add the tags

The existing configuration uses files to store the ‘last commit’. See here:

[dan@devgit-ingress01:/var/db/ingress/repos] $ ls -l
total 28
drwxr-xr-x   7 ingress  ingress  11 Jun  5 18:51 doc
-rw-r--r--   1 ingress  ingress  41 Jun 27 17:21 latest.doc
-rw-r--r--   1 ingress  ingress  41 Jun 27 17:21 latest.ports
-rw-r--r--   1 ingress  ingress  41 Jun 27 17:21 latest.src
drwxr-xr-x  70 ingress  ingress  81 Jun 27 14:51 ports
drwxr-xr-x  27 ingress  ingress  44 Jun 27 01:00 src
[dan@devgit-ingress01:/var/db/ingress/repos] $ 

This is the script I created for that:

#!/bin/sh

# The main branch is always origin/main - so far
refname='origin/main'

repos="doc ports src"
for repo in $repos
do
  tag=$(cat latest.$repo)
  cd $repo
  git tag -m "last known commit of " -f freshports/origin/main $tag
  cd -
done

Running that:

$ # The main branch is always origin/main - so far
$ refname='origin/main'
$ 
$ repos="doc ports src"
$ for repo in $repos
> do
>   tag=$(cat latest.$repo)
>   cd $repo
>   git tag -m "last known commit of " -f freshports/origin/main $tag
>   cd -
> done
/var/db/ingress/repos
/var/db/ingress/repos
/var/db/ingress/repos

Not entirely exciting.

This is the script to display the existing values:

#!/bin/sh

# The main branch is always origin/main - so far
refname='origin/main'

repos="doc ports src"
for repo in $repos
do
  tag=$(cat latest.$repo)
  cd $repo
  git rev-parse --verify freshports/$refname^{}
  cd -
done

Running that:

[dan@devgit-ingress01:/var/db/ingress/repos] $ /bin/sh
$ # The main branch is always origin/main - so far
$ refname='origin/main'
$ 
$ repos="doc ports src"
$ for repo in $repos
> do
>   tag=$(cat latest.$repo)
>   cd $repo
>   git rev-parse --verify freshports/$refname
>   cd ..
> done
048a537b787b9e858df140dfdc2122e3fb49c198
/var/db/ingress/repos
cc60e220aa4fd4c3b0f5eb74b8eb13bcf6f9c406
/var/db/ingress/repos
ca79c1fd81f2b2694a157df3beecb6fb48483900
/var/db/ingress/repos
$ cat latest.doc
4778fc33a57b1a7b9597868fa7fadef63a891edd
$ cat latest.ports
03ed448b501283f1e4d5dd0abb6847a91a32a79b
$ cat latest.src
b762974cf4b9ea77f1decf4a6d829372f0a97f75
$ 

That seems wrong. Why is the hash from the git rev-parse –verify not the same.


EDIT: 2021-07-09 The solution to seeing the actual commit hash is to change the line to:

git rev-parse --verify freshports/$refname^{}

That deferences the tag. The above code has since been updated with that change, for my convenience.

end EDIT.


I spent 30-45 minutes looking in that. Then I did this:

$ git show freshports/origin/main
tag freshports/origin/main
Tagger: Dan Langille <dan@langille.org>
Date:   Sun Jun 27 18:59:35 2021 +0000

last known commit of origin/main

commit 03ed448b501283f1e4d5dd0abb6847a91a32a79b (tag: freshports/origin/main)
Author: Rene Ladan <rene@FreeBSD.org>
Date:   Sun Jun 27 16:56:24 2021 +0200

    sysutils/bhyve-firmware: undeprecate CSM option
    
    The sysutils/uefi-edk2-bhyve port no longer needs Python 2.7

diff --git a/sysutils/bhyve-firmware/Makefile b/sysutils/bhyve-firmware/Makefile
index 1fb78f76049c..3c601a11f7ef 100644
--- a/sysutils/bhyve-firmware/Makefile
+++ b/sysutils/bhyve-firmware/Makefile
@@ -14,11 +14,4 @@ OPTIONS_DEFAULT=     CSM
 CSM_DESC=              Include firmware with Compatibility Support Module
 CSM_RUN_DEPENDS=       uefi-edk2-bhyve-csm>=0:sysutils/uefi-edk2-bhyve-csm
 
-.include <bsd.port.options.mk>

That commit hash matches the value in latest.ports. All is good. This has done the right thing.

Move the script into place

I copied the script from mydev to test-ingress01:

$ scp git-delta.sh devgit-ingress01:
git-delta.sh                                                                                                                                                              

Then moved it into to place:

[dan@devgit-ingress01:/var/db/ingress/repos] $ mv -i ~/git-delta.sh  /usr/local/libexec/freshports/git-delta.sh
[dan@devgit-ingress01:/var/db/ingress/repos] $ sudo pkg check -sr
Checking all packages:  87%
py38-freshports-git-proc-commit-0.0.4_1: checksum mismatch for /usr/local/libexec/freshports/git-delta.sh
Checking all packages: 100%
[dan@devgit-ingress01:/var/db/ingress/repos] $ ls -l /usr/local/libexec/freshports/git-delta.sh
-rwxr-xr-x  1 dan  dan  3977 Jun 27 18:57 /usr/local/libexec/freshports/git-delta.sh

Yep, that’s the right file.

I’m overwriting the file usually installed by package. Most of /usr/local/libexec/freshports/ is part of an svn co of the code. But a few files are not.

[dan@devgit-ingress01:/usr/local/libexec/freshports] $ svn st
?       .idea
?       UpdatePackagesFromRawPackages.py
?       check_freshports_cache_clearing
?       check_git.sh
?       check_report_logs.pl
?       check_repos_for_new_stuff.py
?       fetch-extract-parse-import-one-abi.sh
?       get_packagesite.txz_date
?       git-delta.sh
?       git-range-of-commits.sh
?       git-single-commit.sh
?       git-to-freshports-xml.py
?       helper_scripts
?       import-via-copy-packagesite-all-raw-fields.py
?       import_packagesite.py
?       show-config.sh
?       test-categories.pl
?       test-new-xml-code-single.sh
?       test-new-xml-code-starting-from.sh
?       testing.pm
[dan@devgit-ingress01:/usr/local/libexec/freshports] $ pkg info -l py38-freshports-git-proc-commit 
py38-freshports-git-proc-commit-0.0.4_1:
	/usr/local/libexec/freshports/check_git.sh
	/usr/local/libexec/freshports/git-delta.sh
	/usr/local/libexec/freshports/git-range-of-commits.sh
	/usr/local/libexec/freshports/git-single-commit.sh
	/usr/local/libexec/freshports/git-to-freshports-xml.py
[dan@devgit-ingress01:/usr/local/libexec/freshports] $ 

Now let’s wake up the sleeping ingress daemon.

Trying out the script

I’m going to run the ingress daemon after stopping the freshports daemon. I want to look for incoming commits and create the XML, but I don’t want to process the XML.

$ sudo service freshports stop
Stopping freshports.

Now start:

[dan@devgit-ingress01:~] $ sudo service ingress start
Starting ingress.

And the logs pile up:

[dan@devgit-ingress01:/usr/local/libexec/freshports] $ sudo xtail /var/log/freshports/*.log
^C
*** recently changed files ***
   1  27-Jun-21 19:36:09  /var/log/freshports/ingress-daemon.log
   2  27-Jun-21 19:36:09  /var/log/freshports/freshports.log
   3  27-Jun-21 17:21:06  /var/log/freshports/git.log
   4  27-Jun-21 17:15:11  /var/log/freshports/freshports-daemon.log
   5  27-Jun-21 03:04:23  /var/log/freshports/report-new-ports.daily.log
   6  27-Jun-21 03:04:22  /var/log/freshports/report-notification.daily.log
   7  27-Jun-21 03:03:01  /var/log/freshports/report-security.daily.log
   8  27-Jun-21 00:00:00  /var/log/freshports/report-notification.weekly.log
currently watching:  15 files  0 dirs  0 unknown entries

*** /var/log/freshports/ingress-daemon.log ***
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: starting up!
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: incoming: /var/db/ingress/message-queues/incoming
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: ready
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: yes, there is a job waiting
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: running /usr/local/bin/perl ./job-waiting.pl
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: from directory  /usr/local/libexec/freshports
Jun 27 19:37:34 devgit-ingress01 ingress[74579]: -rw-r--r--  1 dan  dan  2421 May  9 12:36 ./job-waiting.pl

*** /var/log/freshports/freshports.log ***
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: running job-waiting.pl 
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: starting ./job-waiting.pl (/usr/local/libexec/freshports) 
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: running ./job-waiting.pl as user = 'ingress' (/usr/local/libexec/freshports) 
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: checking jobs for ingress (/usr/local/libexec/freshports) 
Jun 27 19:37:34 devgit-ingress01 FreshPorts[74582]: /var/db/ingress/signals/check_git exists.  About to run check_git.sh (/usr/local/libexec/freshports) 
Jun 27 19:37:34 devgit-ingress01 check_git.sh[74584]: /usr/local/libexec/freshports/check_git.sh has started

*** /var/log/freshports/git.log ***
2021.06.27 19:37:34 git-delta.sh has started. Will check these repos: 'doc ports src'
2021.06.27 19:37:34 git-delta.sh XML dir is /var/db/ingress/message-queues/incoming
2021.06.27 19:37:34 git-delta.sh Now processing repo: doc
2021.06.27 19:37:34 git-delta.sh REPODIR='/var/db/ingress/repos/doc' exists
2021.06.27 19:37:34 git-delta.sh Repodir is /var/db/ingress/repos/doc
2021.06.27 19:37:34 git-delta.sh Running: /usr/local/bin/git fetch:
2021.06.27 19:37:34 git-delta.sh Done.
looking at origin/HEAD
looking at origin/main
working on 'origin/main'
1a6d7511cc3a51bdddb132e3e66ee02d6edccb03
2021.06.27 19:37:34 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/main..origin/main
2021.06.27 19:37:34 git-delta.sh Done.
2021.06.27 19:37:34 git-delta.sh No commits were found
2021.06.27 19:37:34 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo doc --path /var/db/ingress/repos/doc ---commit-range /origin/main..origin/main --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: ${freshports/...}: Bad substitution
2021.06.27 19:37:34 git-delta.sh Now processing repo: ports
2021.06.27 19:37:34 git-delta.sh REPODIR='/var/db/ingress/repos/ports' exists
2021.06.27 19:37:34 git-delta.sh Repodir is /var/db/ingress/repos/ports
2021.06.27 19:37:34 git-delta.sh Running: /usr/local/bin/git fetch:
From https://git.FreeBSD.org/ports
   03ed448b5012..16bef8b16b7f  main       -> origin/main
   3a6240c21278..ae162bd98935  2021Q2     -> origin/2021Q2
2021.06.27 19:37:36 git-delta.sh Done.
looking at origin/2014Q1
looking at origin/2014Q2
looking at origin/2014Q3
looking at origin/2014Q4
looking at origin/2015Q1
looking at origin/2015Q2
looking at origin/2015Q3
looking at origin/2015Q4
looking at origin/2016Q1
looking at origin/2016Q2
looking at origin/2016Q3
looking at origin/2016Q4
looking at origin/2017Q1
looking at origin/2017Q2
looking at origin/2017Q3
looking at origin/2017Q4
looking at origin/2018Q1
looking at origin/2018Q2
looking at origin/2018Q3
looking at origin/2018Q4
looking at origin/2019Q1
looking at origin/2019Q2
looking at origin/2019Q3
looking at origin/2019Q4
looking at origin/2020Q1
looking at origin/2020Q2
looking at origin/2020Q3
looking at origin/2020Q4
looking at origin/2021Q1
looking at origin/2021Q2
working on 'origin/2021Q2'
'git tag -l freshports/origin/2021Q2' found nothing
Let's find the first commit in this branch
First ref is '4e3cf0163c4a00d4dac41d6da43472d2fcab2f29'
taging that now:
2021.06.27 19:37:36 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/2021Q2..origin/2021Q2
2021.06.27 19:37:36 git-delta.sh Done.
2021.06.27 19:37:36 git-delta.sh The commits found are:
2021.06.27 19:37:36 git-delta.sh ae162bd989359e2e599a2b9cb58da87bdec05fab
2021.06.27 19:37:36 git-delta.sh 3a6240c2127867111d6eb18e39f539e35e3ffad7

[redacted for brevity]

2021.06.27 19:37:37 git-delta.sh 144bb249fc6f49387b8e0ba2cd27ef5f61430224
2021.06.27 19:37:37 git-delta.sh 5fbb95157840f744df004acd2e9a56b68422a632
2021.06.27 19:37:37 git-delta.sh ba68dd511d88a2fb6c4fefb53bc401b0f86e3217
2021.06.27 19:37:37 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo ports --path /var/db/ingress/repos/ports ---commit-range /origin/2021Q2..origin/2021Q2 --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: ${freshports/...}: Bad substitution
2021.06.27 19:37:37 git-delta.sh Now processing repo: src
2021.06.27 19:37:37 git-delta.sh REPODIR='/var/db/ingress/repos/src' exists
2021.06.27 19:37:37 git-delta.sh Repodir is /var/db/ingress/repos/src
2021.06.27 19:37:37 git-delta.sh Running: /usr/local/bin/git fetch:
From https://git.FreeBSD.org/src
   b762974cf4b..c7f048ab353  main       -> origin/main
2021.06.27 19:37:38 git-delta.sh Done.
looking at origin/HEAD
looking at origin/main
working on 'origin/main'
cd8b62fb8c563b11dc6b7562b6a604258e73161b
2021.06.27 19:37:38 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/main..origin/main
2021.06.27 19:37:38 git-delta.sh Done.
2021.06.27 19:37:38 git-delta.sh The commits found are:
2021.06.27 19:37:38 git-delta.sh c7f048ab3532a9f081addd6da0adf96f25271de8
2021.06.27 19:37:38 git-delta.sh 914b6a0122c3032c2c2ae3a44bdeaf69e37efd61
2021.06.27 19:37:38 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py --repo src --path /var/db/ingress/repos/src ---commit-range /origin/main..origin/main --spooling /var/db/ingress/message-queues/spooling --output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: ${freshports/...}: Bad substitution
2021.06.27 19:37:38 git-delta.sh Ending

*** /var/log/freshports/ingress-daemon.log ***
Jun 27 19:37:38 devgit-ingress01 ingress[74579]: job-waiting.pl finishes normally

*** /var/log/freshports/freshports.log ***
Jun 27 19:37:38 devgit-ingress01 check_git.sh[75322]: /usr/local/libexec/freshports/check_git.sh has finished
Jun 27 19:37:38 devgit-ingress01 FreshPorts[74582]: Finished running check_git.sh (/usr/local/libexec/freshports) 
Jun 27 19:37:38 devgit-ingress01 FreshPorts[74582]: flag '/var/db/ingress/signals/check_git' not set.  no work for check_git.sh (/usr/local/libexec/freshports) 

The main issue I see is the Bad substitution errors on line 47, 104, and 121. If you look back at lines 115-118 at the bottom of Processing commits on branches with git – part 2. In short:

${xfreshports}/${refname}..$refname

Became

freshports/${refname}..${refname}

I also see that I need to set the tag freshports/origin/2021Q2 on the ports repo (see line 87 above). I’ve already processed some of those commits last weekend. Looking at https://devgit.freshports.org/?branch=quarterly I see the last commit (as of when I type this) is 5ceea22.

This is how I set that tag:

[dan@devgit-ingress01:~] $ sudo su -l ingress
$ bash
[ingress@devgit-ingress01 ~]$ cd repos/ports
[ingress@devgit-ingress01 ~/repos/ports]$ git tag -m "last known commit of freshports/origin/2021Q2" -f freshports/origin/2021Q2 5ceea227c504d2892d91c1aa8d8d81ff15b22fc3
Updated tag 'freshports/origin/2021Q2' (was dbbd50cc2513)
[ingress@devgit-ingress01 ~/repos/ports]$ 

The second run

On the second run, the found commits looked good, but git-to-freshports-xml.py was bringing in port commits from 1994-08-21, doc commits from 1995-08-27, and src commits from 1993-06-12. In short, it was building XML files for the entire repo. I don’t want that, not right now. Someone day we can look at importing all commits, but I suspect there will be infrastructure changes which are not backwards compatible with port commits.

Now I’m into debugging and I won’t post all of this work.

I think /usr/local/libexec/freshports/git-to-freshports-xml.py needs to be updated.

Jun 272021
 

Following on from last weekend’s work, today I’m learning more about git, and the difference between Plumbing and Porcelain commands.

The original intent was to use git tag -l freshports/$refname to detect if a tag exists. The problem there, it always returns 0.

[dan@mydev:/var/db/ingress/repos/ports] $ git tag -l | head
10-eol
4-eol
5-eol
6-eol
7-eol
8-eol
9-eol
pkg-install-eol
pre-xorg-7
release/10.0.0
[dan@mydev:/var/db/ingress/repos/ports] $ 


[dan@mydev:/var/db/ingress/repos/ports] $ git tag -l 10-eol
10-eol
[dan@mydev:/var/db/ingress/repos/ports] $ echo $?
0
[dan@mydev:/var/db/ingress/repos/ports] $ git tag -l 10-eollkjadslkjdf
[dan@mydev:/var/db/ingress/repos/ports] $ echo $?
0
[dan@mydev:/var/db/ingress/repos/ports] $ git tag -l | grep 10-eollkjadslkjdf
[dan@mydev:/var/db/ingress/repos/ports] $ 

From the above, whether the tag exists or not, the code can’t tell, well, without looking at the output.

Let’s use this instead:

[dan@mydev:/var/db/ingress/repos/ports] $ git rev-parse --verify refs/tags/10-eol
1aa5d44ff272774be87b1afa6a71fa6cbd62c8af
[dan@mydev:/var/db/ingress/repos/ports] $ echo $?
0
[dan@mydev:/var/db/ingress/repos/ports] $ git rev-parse --verify refs/tags/10-eollkjadslkjdf
fatal: Needed a single revision
[dan@mydev:/var/db/ingress/repos/ports] $ echo $
[dan@mydev:/var/db/ingress/repos/ports] $ 

We can also use the -q to suppress output.

Next goal

Let’s try this on the devgit.freshports.org server next.

This is the current code:

#!/bin/sh

# process the new commits
# based upon https://github.com/FreshPorts/git_proc_commit/issues/3
# An idea from https://github.com/sarcasticadmin

if [ ! -f /usr/local/etc/freshports/config.sh ]
then
	echo "/usr/local/etc/freshports/config.sh.sh not found by $0"
	exit 1
fi

# this can be a space separated list of repositories to check
# e.g. "doc ports src"
repos=$1

. /usr/local/etc/freshports/config.sh

LOGGERTAG='git-delta.sh'

logfile "has started. Will check these repos: '${repos}'"

# what remote are we using on this repo?
REMOTE='origin'

# where we do dump the XML files which we create?
XML="${INGRESS_MSGDIR}/incoming"

logfile "XML dir is $XML"

for repo in ${repos}
do
   logfile "Now processing repo: ${repo}"

   # convert the repo label to a physical directory on disk
   dir=$(convert_repo_label_to_directory ${repo})

   # empty means error
   if [  "${dir}" == "" ]; then
      logfile "FATAL error, repo='${repo}' is unknown: cannot translate it to a directory name"
      continue
   fi

   # where is the repo directory?
   # This is the directory which contains the repos.
   REPODIR="${INGRESS_PORTS_DIR_BASE}/${dir}"
   LATEST_FILE="${INGRESS_PORTS_DIR_BASE}/latest.${dir}"

   if [ -d ${REPODIR} ]; then
      logfile "REPODIR='${REPODIR}' exists"
   else
      logfile "FATAL error, REPODIR='${REPODIR}' is not a directory"
      continue
   fi

   logfile "Repodir is $REPODIR"
   # on with the work

   cd ${REPODIR}

   # Bring local branch up-to-date with the local remote
   logfile "Running: ${GIT} fetch:"
   ${GIT} fetch
   logfile "Done."

   NAME_OF_HEAD="main"
   NAME_OF_REMOTE="origin"
   MAIN_BRANCH="$NAME_OF_REMOTE/$NAME_OF_HEAD"

   git for-each-ref --format '%(objecttype) %(refname)' \
      | sed -n 's/^commit refs\/remotes\///p' \
      | while read -r refname
   do
      echo looking at $refname
      # for now, when testing, only this branch please
      if [ "$refname" != "origin/2021Q2" ] && [ "$refname" != "$MAIN_BRANCH" ]
      then
         continue
      fi

      echo "working on '$refname'"

      if ! git rev-parse -q --verify freshports/$refname
      then
         if [ "$refname" == "$MAIN_BRANCH" ]
         then
            echo "FATAL - '$MAIN_BRANCH' must have tag 'freshports/$refname' set manually  - special case the main branch because the best merge base is the most recent commit"
            exit
         fi
         echo "'git tag -l freshports/$refname'" found nothing
         echo "Let's find the first commit in this branch"
         first_ref=$(git merge-base $NAME_OF_REMOTE/$NAME_OF_HEAD $refname)
         echo "First ref is '$first_ref'"
         # get the first commit of that branch and create a tag.
         echo taging that now:
         git tag -m "first known commit of $refname" -f freshports/$refname $first_ref
      fi

      # get list of commits, if only to document them here
      logfile "Running: ${GIT} rev-list freshports/$refname..$refname"
      commits=$(${GIT} rev-list freshports/$refname..$refname)
      logfile "Done."

      if [ -z "$commits" ]
      then
         logfile "No commits were found"
      else
        logfile "The commits found are:"
        for commit in $commits
        do
           logfile "$commit"
        done
      fi

      logfile "${SCRIPTDIR}/git-to-freshports-xml.py --repo ${repo} --path ${REPODIR} ---commit-range ${xfreshports}/${refname}..$refname --spooling ${INGRESS_SPOOLINGDIR} --output ${XML}"
      
      
               ${SCRIPTDIR}/git-to-freshports-xml.py --repo ${repo} --path ${REPODIR} ---commit-range ${freshports/$refname}..$refname --spooling ${INGRESS_SPOOLINGDIR} --output ${XML}

      new_latest=$(${GIT}  rev-parse HEAD)
      
      echo new_latest = $new_latest

      # echo $new_latest > ${LATEST_FILE}
      # Store the last known commit that we just processed.
      git tag -m "last known commit of $refname" -f freshports/$refname $refname

   done
done

logfile "Ending"
Jun 262021
 

When upgrading packages on a FreshPorts ingress node (that is usually a jail which processes incoming commits and loads them into the database), I believe there is a possibility of interrupting commit processing. It’s a race condition between pkg removing a file and that file being required by the commit processing.

This post serves to document one way of avoiding this.

The prcocesses

These are expected processes running in the testgit ingress node:

[dan@testgit-ingress01:~] $ ps auwwx
USER         PID %CPU %MEM   VSZ   RSS TT  STAT STARTED    TIME COMMAND
dan        61389  0.2  0.0 13248  4392  1  SJ   17:12   0:00.01 bash
root       61379  0.1  0.0 20632  9600  -  SsJ  17:12   0:00.02 sshd: dan [priv] (sshd)
dan        61382  0.1  0.0 20636  9616  -  SJ   17:12   0:00.01 sshd: dan@pts/1 (sshd)
dan        61383  0.1  0.0 12200  3340  1  SsJ  17:12   0:00.01 -sh (sh)
nagios      8455  0.0  0.0 15960  6392  -  IsJ  11Jun21 0:21.49 /usr/local/sbin/nrpe3 -c /usr/local/etc/nrpe.cfg -d
postfix    22888  0.0  0.0 51464 10744  -  IJ   16:27   0:00.02 pickup -l -t unix -u
freshports 23285  0.0  0.0 11004  2424  -  IsJ  18:52   0:00.04 daemon: freshports[23286] (daemon)
freshports 23286  0.0  0.0 11656  3048  -  SJ   18:52   0:10.78 /bin/sh /usr/local/libexec/freshports-service/freshports.sh
root       42995  0.0  0.0 11376  2772  -  SsJ   2Jun21 0:41.85 /usr/sbin/syslogd -s
root       43331  0.0  0.0 51516  8784  -  IsJ   2Jun21 0:34.10 /usr/local/libexec/postfix/master -w
postfix    43333  0.0  0.0 51568  8836  -  IJ    2Jun21 0:04.54 qmgr -l -t unix -u
ingress    43356  0.0  0.0 11004  2424  -  IsJ   2Jun21 0:01.97 daemon: ingress[43357] (daemon)
ingress    43357  0.0  0.0 11576  2752  -  SJ    2Jun21 1:44.53 /bin/sh /usr/local/libexec/freshports-service/ingress.sh
root       43390  0.0  0.0 19916  8792  -  SsJ   2Jun21 0:07.48 /usr/sbin/sshd
root       43402  0.0  0.0 11308  2632  -  IsJ   2Jun21 0:16.39 /usr/sbin/cron -s
postfix    44636  0.0  0.0 52156 11140  -  IJ    2Jun21 0:02.75 tlsmgr -l -t unix -u
postfix    53480  0.0  0.0 51588 10760  -  IJ   17:09   0:00.01 anvil -l -t unix -u
ingress    61381  0.0  0.0 10676  2188  -  SCJ  17:12   0:00.00 sleep 3
freshports 61430  0.0  0.0 10676  2188  -  SCJ  17:12   0:00.00 sleep 3
dan        61431  0.0  0.0 11704  3064  1  R+J  17:12   0:00.00 ps auwwx
[dan@testgit-ingress01:~] $ 

Of note, nrpe is running in the jail. I suspect I can run that only on the host and inspect the jail.

Some may claim postfix is overkill for running on here, but I have yet to run options which satisfy my [unstated, yes] requirements. But it’s a wonderful MTA and I know how to use it.

There, now I have two upcoming projects:

  1. replace postfix with something smaller on ingress and web nodes
  2. move nrpe from jails to hosts

Stop commit processing

To stop commit the discovery of new commits:

$ sudo service ingress stop
Stopping ingress.

This is a shell script (/usr/local/libexec/freshports-service/ingress.sh): it was carrying out any effective work, it would be interrupted. I don’t know how to properly shutdown this script. In general, it gets a new job created every three minutes, on the 3’s, via a periodic script (see /usr/local/etc/periodic/everythreeminutes/215.fp_check_git_for_commits). If you’re not near a multiple of 3 minutes, and all you see running is the above, I think it’s safe to stop ingress.

To stop the processing of discovered commits (i.e. the output from the ingress daemon):

$ sudo service freshports stop
Stopping freshports.

Now you should see:

 ps auwwx
USER      PID %CPU %MEM   VSZ   RSS TT  STAT STARTED    TIME COMMAND
nagios   8455  0.0  0.0 15960  6392  -  IsJ  11Jun21 0:21.50 /usr/local/sbin/nrpe3 -c /usr/local/etc/nrpe.cfg -d
postfix 22888  0.0  0.0 51464 10744  -  IJ   16:27   0:00.02 pickup -l -t unix -u
root    42995  0.0  0.0 11376  2772  -  SsJ   2Jun21 0:41.87 /usr/sbin/syslogd -s
root    43331  0.0  0.0 51516  8784  -  IsJ   2Jun21 0:34.12 /usr/local/libexec/postfix/master -w
postfix 43333  0.0  0.0 51568  8836  -  IJ    2Jun21 0:04.54 qmgr -l -t unix -u
root    43390  0.0  0.0 19916  8792  -  IsJ   2Jun21 0:07.48 /usr/sbin/sshd
root    43402  0.0  0.0 11308  2632  -  IsJ   2Jun21 0:16.40 /usr/sbin/cron -s
postfix 44636  0.0  0.0 52156 11140  -  IJ    2Jun21 0:02.75 tlsmgr -l -t unix -u
root    61379  0.0  0.0 20632  9600  -  IsJ  17:12   0:00.02 sshd: dan [priv] (sshd)
dan     61382  0.0  0.0 20636  9624  -  SJ   17:12   0:00.05 sshd: dan@pts/1 (sshd)
postfix 84985  0.0  0.0 51588 10760  -  IJ   17:24   0:00.01 anvil -l -t unix -u
dan     61383  0.0  0.0 12200  3340  1  IsJ  17:12   0:00.01 -sh (sh)
dan     61389  0.0  0.0 13256  4396  1  SJ   17:12   0:00.04 bash
dan     92797  0.0  0.0 11696  3060  1  R+J  17:27   0:00.00 ps auwwx

Periodic scripts

All periodic scripts should should observe the OFFLINE variable defined in /usr/local/etc/freshports/config.sh. This variable is set depending on whether or not /usr/local/etc/freshports/OFFLINE exists.

So to effectively block those scripts, issue this command:

$ sudo touch /usr/local/etc/freshports/OFFLINE

After the upgrade

After the upgrade, issue these commands:

$ sudo rm /usr/local/etc/freshports/OFFLINE
$ sudo service freshports start
$ sudo service ingress start

Not perfect

I am sure this is not perfect, but is better than not doing it. It is also best not to do maintenance during these windows:

  1. the top of the hour, when the hourly periodic scripts run.
  2. Just after 0301 UTC when the daily periodic scripts run.
  3. Just after 0415 UTC Saturdays when the weekly periodic scripts run.
  4. Just after 0530 UTC on the first of the month when the monthly periodic scripts run.

I tend to run updates between 1200 and 2000 UTC for personal convenience.

Jun 212021
 

FreshPorts has three main daemons:

  1. ingress – waits for incoming commits and creates XML.
  2. freshports – processes the XML into the database.
  3. fp-listen – listens for events and clears cache

The first two run on the backend (the ingress node, or jail), the last run runs on the webserver (nginx jail).

The list of events

The first thing fp-listen does is read the list of events from the database:

freshports.devgit=# select * from listen_for;
 id |       name        |       script_name        
----+-------------------+--------------------------
  5 | port_updated      | listen_port
  6 | ports_moved       | listen_ports_moved
  7 | ports_updating    | listen_ports_updating
  8 | vuxml             | listen_vuxml
 10 | category_new      | listen_category_new
 11 | date_updated      | listen_date_updated
 12 | packages_imported | listen_packages_imported
 13 | listen_for        | ClearPackagesCache
(8 rows)

Let’s add our new event and the name of the script we-have-yet-to-write:

freshports.devgit=# insert into listen_for (name, script_name) values ('commit_updated', 'listen_commit');
INSERT 0 1

However, note that I won’t actually be created a script by that name.

The Python code

Here is the Python code:

[dan@devgit-nginx01:~/src/fp-listen/fp-listen] $ svn di
Index: fp-listen.py
===================================================================
--- fp-listen.py	(revision 5642)
+++ fp-listen.py	(working copy)
@@ -110,6 +110,44 @@
 
   syslog.syslog(syslog.LOG_NOTICE, "Done with PackagesCacheClear()");
 
+def CommitsCacheClear():
+  dbh = psycopg2.connect(DSN)
+  curs = dbh.cursor(cursor_factory=psycopg2.extras.DictCursor)
+
+  curs.execute("SELECT commit_to_clear FROM cache_clearing_commits")
+  NumRows = curs.rowcount
+  dbh.commit();
+  if (NumRows > 0):
+    syslog.syslog(syslog.LOG_NOTICE, 'COUNT: %d entries to process' % (NumRows))
+    rows = curs.fetchall()
+    for row in rows:
+      #
+      filenameglob = config['dirs']['COMMIT_CACHE_PATH'] % (row['commit_to_clear'])
+      syslog.syslog(syslog.LOG_NOTICE, 'removing glob %s' % (filenameglob))
+
+      try:
+        for filename in glob.glob(filenameglob):
+          syslog.syslog(syslog.LOG_NOTICE, 'removing %s' % (filename))
+          if os.path.isfile(filename):
+            os.remove(filename)
+          else:
+            shutil.rmtree(filename)
+
+      except FileNotFoundError:
+        syslog.syslog(syslog.LOG_CRIT, 'could not find file for deletion %s' % (filename))
+
+      syslog.syslog(syslog.LOG_NOTICE, "DELETE FROM cache_clearing_commits WHERE commit_to_clear = '%s'" % (row['commit_to_clear']))
+      curs.execute("DELETE FROM cache_clearing_commits WHERE commit_to_clear = '%s'" % (row['commit_to_clear']))
+      dbh.commit()
+
+    # end for
+  else:
+    syslog.syslog(syslog.LOG_ERR, 'ERROR: No cached entries found for removal')
+  # end if
+    
+  syslog.syslog(syslog.LOG_NOTICE, 'finished')
+  return NumRows
+
 def ClearDateCacheEntries():
   syslog.syslog(syslog.LOG_NOTICE, 'checking for cache date to remove...')
   dbh = psycopg2.connect(DSN)
@@ -154,6 +192,7 @@
   syslog.syslog(syslog.LOG_NOTICE, 'finished')
   return NumRows
 
+
 def Touch(File):
   if not os.path.exists(File):
     fd = open(File, 'aos.O_WRONLY | os.O_NONBLOCK | os.O_CREAT | os.O_NOCTTY | os.O_APPEND')
@@ -160,6 +199,7 @@
     fd.close()
   os.utime(File, None)
 
+
 def ProcessCategoryNew():
   syslog.syslog(syslog.LOG_NOTICE, 'We have a new category')
   
@@ -167,14 +207,18 @@
   Touch(config['flags']['WWWENPortsCategoriesFlag'])
   Touch(config['flags']['JOBWAITING'])
 
+
 def ProcessPortsMoved():
   syslog.syslog(syslog.LOG_NOTICE, 'processing ports/MOVED')
 
+
 def ProcessPortsUpdating():
   syslog.syslog(syslog.LOG_NOTICE, 'processing ports/UPDATING')
 
+
 def ProcessVUXML():
   syslog.syslog(syslog.LOG_NOTICE, 'processing ports/security/portaudit/vuln.xml')
+
   
 def ClearMiscCaches():
   syslog.syslog(syslog.LOG_NOTICE, 'invoked: ClearMiscCaches()');
@@ -261,6 +305,9 @@
           # at the time of writing, there was no reason to ClearMiscCaches() when
           # new packages arrive
           clear_cache = False;
+        elif listens[notify.channel] == 'listen_commit':
+          syslog.syslog(syslog.LOG_NOTICE, "invoking CommitsClearCache()");
+          CommitsCacheClear()
         else:
           clear_cache = False;
           syslog.syslog(syslog.LOG_ERR, "Code does not know what to do when '%s' is found." % notify.channel)
[dan@devgit-nginx01:~/src/fp-listen/fp-listen] $ 

Directory permissions

When doing the initial testing, I saw this error:

Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: Traceback (most recent call last):
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:   File "/usr/local/lib/python3.8/site-packages/fp-listen/fp-listen.py", line 310, in 
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:     CommitsCacheClear()
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:   File "/usr/local/lib/python3.8/site-packages/fp-listen/fp-listen.py", line 134, in CommitsCacheClear
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:     shutil.rmtree(filename)
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:   File "/usr/local/lib/python3.8/shutil.py", line 722, in rmtree
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:     onerror(os.rmdir, path, sys.exc_info())
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:   File "/usr/local/lib/python3.8/shutil.py", line 720, in rmtree
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:     os.rmdir(path)
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: PermissionError: [Errno 13] Permission denied: '/var/db/freshports/cache/commits/400c1af36e44f6ad699125526c70668745482c98/'

Ahh, that’s because we have this:

[dan@devgit-nginx01:/usr/local/etc/freshports] $ ls -l /var/db/freshports/cache
total 432
drwxr-xr-x  2 www         freshports       2 Feb 28 18:20 categories
drwxr-xr-x  3 www         freshports       3 Jun 21 23:54 commits
drwxrwxr-x  3 www         freshports       3 Jun 20 15:51 daily
drwxr-xr-x  2 www         freshports       2 Feb 28 18:20 general
drwxrwxr-x  2 freshports  freshports       9 Jun 21 23:00 html
drwxrwxr-x  2 www         freshports       2 Jun 21 23:47 news
drwxrwxr-x  2 www         freshports       2 Feb 28 18:20 packages
drwxr-xr-x  2 www         freshports       2 Feb 28 18:20 pages
drwxrwxr-x  6 www         freshports       6 Jun 20 14:30 ports
-rw-r--r--  1 www         freshports  414331 Jun 13 20:41 searchlog.txt
drwxr-xr-x  2 www         freshports       3 Jun 21 23:54 spooling

Ahh, freshports can’t write to the commits directory, so it cannot delete files. The fp-listen daemon is running as the freshports user:

$ ps auwwx | grep fp-listen
freshports 44828  0.0  0.0  29756 19472  -  SJ   00:06   0:00.19 /usr/local/bin/python /usr/local/lib/python3.8/site-packages/fp-listen/fp-listen.py (python3.8)
dan        77297  0.0  0.0  11456  2776  6  S+J  00:22   0:00.00 grep fp-listen

I can’t just chmod +w here. There is a zfs snapshot involved. Here is what I did instead:

$ sudo zfs rollback nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits@empty
$ sudo zfs destroy nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits@empty
$ sudo chmod g+w commits
$ sudo zfs snapshot nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits@empty

The destroy had to be done on the host itself. This is a situation I’ve seen before where a snapshot cannot be deleted from within the jail, but it can be created.

Configuration file changes

This new entry is required in /usr/local/etc/freshports/fp-listen.ini:

[dan@devgit-nginx01:~/src/fp-listen] $ svn di etc/freshports/fp-listen.ini.sample
Index: etc/freshports/fp-listen.ini.sample
===================================================================
--- etc/freshports/fp-listen.ini.sample	(revision 5642)
+++ etc/freshports/fp-listen.ini.sample	(working copy)
@@ -13,6 +13,7 @@
 BASEDIR         = /var/db/freshports
 
 CATEGORY_CACHE_PATH = %(BASEDIR)s/cache/categories/%%s/*
+COMMIT_CACHE_PATH   = %(BASEDIR)s/cache/commits/%%s/
 PORT_CACHE_PATH     = %(BASEDIR)s/cache/ports/%%s/%%s/*
 DATE_CACHE_PATH     = %(BASEDIR)s/cache/daily/%%s/%%s/%%s.*
 NEWS_CACHE_DIR      = %(BASEDIR)s/cache/news/
[dan@devgit-nginx01:~/src/fp-listen] $ 
Jun 212021
 

Following on from yesterdays Quarterly branches: what’s next? and Processing commits on branches with git, I’ve started in on clearing commits from cache.

The code

First step, create a table which looks like this:

freshports.devgit=# \d cache_clearing_commits
                     Table "public.cache_clearing_commits"
     Column      |            Type             | Collation | Nullable | Default 
-----------------+-----------------------------+-----------+----------+---------
 commit_to_clear | text                        |           | not null | 
 date_added      | timestamp without time zone |           |          | now()
Indexes:
    "cache_clearing_commits_idx" PRIMARY KEY, btree (commit_to_clear)

freshports.devgit=# 

Then, adjust some existing triggers to do something like this:

+   INSERT INTO cache_clearing_commits (commit_to_clear) VALUES (OLD.message_id)
+      ON CONFLICT ON CONSTRAINT cache_clearing_commits_idx DO NOTHING;
+
+   GET DIAGNOSTICS l_row_count = ROW_COUNT;
+   IF l_row_count > 0 THEN
+      NOTIFY commit_updated;
+   END IF;
+

At first, I started created new triggers etc, but then I looked at the existing triggers on the commit_log table:

Triggers:
    commit_log_delete_check BEFORE DELETE ON commit_log FOR EACH ROW EXECUTE FUNCTION commit_log_delete_check()
    commit_log_insert AFTER INSERT ON commit_log FOR EACH ROW EXECUTE FUNCTION commit_log_insert()
    commit_log_update AFTER UPDATE ON commit_log FOR EACH ROW EXECUTE FUNCTION commit_log_update()

I removed what I created and moved that code into the existing triggers.

The test

Let’s do a simple non-destructive test:

freshports.devgit=# begin;
BEGIN
freshports.devgit=# update commit_log set message_id = message_id where message_id = '524260db7683681c7deec9f1968c15a717317685';
UPDATE 1
freshports.devgit=# select * from cache_clearing_commits;
             commit_to_clear              |         date_added         
------------------------------------------+----------------------------
 524260db7683681c7deec9f1968c15a717317685 | 2021-06-21 22:46:35.674694
(1 row)

freshports.devgit=# 

It works.

Next step: modify the fp-listen daemon to listen for commit_updated.

Jun 202021
 

With today’s work on Processing commits on branches with git, good progress was made.

  1. You can see commits on the quarterly branch (not up to date, because it’s not yet automated).
  2. You can see the port on the quarterly branch, if there’s a commit on the branch.
  3. Want to see a particular commit on a quarterly branch? You can.

What’s next?

Automate what I did

I did some coding, then jumped ahead to creation of the XML file.

I have to go back and continue that code. I suspect I’ll need to special case some branches, mainly main.

Clear cache for commits

When I test, I run a commit, delete it, run again. If that commit is cached by the front end, I have to manually delete that. I don’t want to do that any more. I want to add a trigger which raises a NOTIFY event, which the front end will use to clear the cache. See issue 75.

Edit: 2021-06-21 – done. See Commit cache clearing and adding a new event to fp-listen.

See also issue 75.

Fix caching of commits

When you look at a commit on a branch, it says: Commit found by commit id on branch 2021Q2. That is, if you’re the first one to look at it. That gets cached as Commit found by commit id on branch (no branch value shown). This is a known issue.

Edit: 2021-06-22 – done via this commit.

Fix links on quarterly branch commits

See gitea update for an example.

See issue 163

Edit: 2021-06-24 – done via a72b4a9.

Edit: 2021-06-25 – all of the work above (except the automation) is now in production.

Jun 202021
 

Transitioning from subversion to git was both technically and personally challenging. During the COVID-19 pandemic there was a lot of work to carry out. A new server was created on AWS with a new layout and structure. For a few months it was nothing but my day job and FreshPorts in all my non-working hours.

This post was followed by Processing commits on branches with git – part 2.

The new behind-the-scenes code works better than the subversion code, mostly because it no longer relies upon email for notifications of new commits to the code.

One aspect of commit processing not yet tackled is commits on branches.

Back in November 2020, Mathieu Arnold wrote about a proposed approach. This is what I’m going to start on today.

From what I understood, freshports never needs to be able to access
files on the top of a branch, it only needs to access the files on
specific commits (the fact that a commit is at the top of the branch is
only an artefact of the process). So, you never need to checkout any
branches, you only need to checkout commits. So you never have a HEAD
that points to a branch, HEAD is always in a detached state and points
to a commit.

So what you need to do is, git clone the repository, and then, each time
the script runs, do (in somewhat shell script):

NAME_OF_REMOTE=origin
NAME_OF_HEAD=main

cd /where/the/repo/is

git fetch

# get all references (so, branches, tags, and so one) and keep only
# branches (named commits here) that the remote repository has

git for-each-ref --format '%(objecttype) %(refname)' \
  | sed -n 's/^commit refs\/remotes\///p'
  | while read -r type refname
do

  # If we don't have the tag, it means we have not encountered the
  # branch yet.
  if ! git tag -l freshports/$refname
  then

    # get the first commit of that branch and create a tag.
    git tag -m "first known commit of $refname" -f freshports/$refname $(git merge-base $NAME_OF_REMOTE/$NAME_OF_HEAD $refname)
  fi

  # Get the list of commits between the last known one and the tip of
  # the branch, list may be empty.
  git rev-list freshports/$refname..$refname | while read commithash
  do
    # checkout that commit (with -f so that if some file got changed, we
    # overwrite everything
    git checkout -f $commithash

    # process the commit
  done

  # Store the last known commit that we just processed.
  git tag -m "last known commit of $refname" -f freshports/$refname $refname
done

That’s all, you never need to merge or pull or whatever else.

My starting point, the existing git-delta.sh script, which obtains a list of commits which have occurred after a given commit.

Get the repo

Here, I get the ports repo.

mkdir -p ~/src/repos
git clone https://git.FreeBSD.org/ports.git
cd ports

First attempt

[dan@mydev:~/src/repos/ports] $ git for-each-ref --format '%(objecttype) %(refname)' 
commit refs/heads/2021Q2
commit refs/heads/main
commit refs/remotes/origin/2014Q1
commit refs/remotes/origin/2014Q2
commit refs/remotes/origin/2014Q3
commit refs/remotes/origin/2014Q4
commit refs/remotes/origin/2015Q1
commit refs/remotes/origin/2015Q2
commit refs/remotes/origin/2015Q3
commit refs/remotes/origin/2015Q4
commit refs/remotes/origin/2016Q1
commit refs/remotes/origin/2016Q2
commit refs/remotes/origin/2016Q3
commit refs/remotes/origin/2016Q4
commit refs/remotes/origin/2017Q1
commit refs/remotes/origin/2017Q2
commit refs/remotes/origin/2017Q3
commit refs/remotes/origin/2017Q4
commit refs/remotes/origin/2018Q1
commit refs/remotes/origin/2018Q2
commit refs/remotes/origin/2018Q3
commit refs/remotes/origin/2018Q4
commit refs/remotes/origin/2019Q1
commit refs/remotes/origin/2019Q2
commit refs/remotes/origin/2019Q3
commit refs/remotes/origin/2019Q4
commit refs/remotes/origin/2020Q1
commit refs/remotes/origin/2020Q2
commit refs/remotes/origin/2020Q3
commit refs/remotes/origin/2020Q4
commit refs/remotes/origin/2021Q1
commit refs/remotes/origin/2021Q2
commit refs/remotes/origin/HEAD
commit refs/remotes/origin/main
tag refs/tags/10-eol
tag refs/tags/4-eol
tag refs/tags/5-eol
tag refs/tags/6-eol
tag refs/tags/7-eol
tag refs/tags/8-eol
tag refs/tags/9-eol
tag refs/tags/pkg-install-eol
tag refs/tags/pre-xorg-7
tag refs/tags/release/10.0.0
tag refs/tags/release/10.1.0
tag refs/tags/release/10.2.0
tag refs/tags/release/10.3.0
tag refs/tags/release/10.4.0
tag refs/tags/release/11.0.0
tag refs/tags/release/11.1.0
tag refs/tags/release/11.2.0
tag refs/tags/release/11.3.0
tag refs/tags/release/11.4.0
tag refs/tags/release/12.0.0
tag refs/tags/release/12.1.0
tag refs/tags/release/12.2.0
tag refs/tags/release/13.0.0
tag refs/tags/release/2.0.5
tag refs/tags/release/2.0.5a
tag refs/tags/release/2.1.0
tag refs/tags/release/2.1.5
tag refs/tags/release/2.1.6
tag refs/tags/release/2.1.7
tag refs/tags/release/2.2.0
tag refs/tags/release/2.2.1
tag refs/tags/release/2.2.2
tag refs/tags/release/2.2.5
tag refs/tags/release/2.2.6
tag refs/tags/release/2.2.7
tag refs/tags/release/2.2.8
tag refs/tags/release/3.0.0
tag refs/tags/release/3.1.0
tag refs/tags/release/3.2.0
tag refs/tags/release/3.3.0
tag refs/tags/release/3.4.0
tag refs/tags/release/3.5.0
tag refs/tags/release/4.0.0
tag refs/tags/release/4.1.0
tag refs/tags/release/4.1.1
tag refs/tags/release/4.10.0
tag refs/tags/release/4.11.0
tag refs/tags/release/4.2.0
tag refs/tags/release/4.3.0
tag refs/tags/release/4.4.0
tag refs/tags/release/4.5.0
tag refs/tags/release/4.6.0
tag refs/tags/release/4.6.1
tag refs/tags/release/4.6.2
tag refs/tags/release/4.7.0
tag refs/tags/release/4.8.0
tag refs/tags/release/4.9.0
tag refs/tags/release/5.0.0
tag refs/tags/release/5.1.0
tag refs/tags/release/5.2.0
tag refs/tags/release/5.2.1
tag refs/tags/release/5.3.0
tag refs/tags/release/5.4.0
tag refs/tags/release/5.5.0
tag refs/tags/release/6.0.0
tag refs/tags/release/6.1.0
tag refs/tags/release/6.2.0
tag refs/tags/release/6.3.0
tag refs/tags/release/6.4.0
tag refs/tags/release/7.0.0
tag refs/tags/release/7.1.0
tag refs/tags/release/7.2.0
tag refs/tags/release/7.3.0
tag refs/tags/release/7.4.0
tag refs/tags/release/8.0.0
tag refs/tags/release/8.1.0
tag refs/tags/release/8.2.0
tag refs/tags/release/8.3.0
tag refs/tags/release/8.4.0
tag refs/tags/release/9.0.0
tag refs/tags/release/9.1.0
tag refs/tags/release/9.2.0
tag refs/tags/release/9.3.0
[dan@mydev:~/src/repos/ports] $ 
Raw

Applying the filter

My first try brought back empty strings:

[dan@mydev:~/src/repos/ports] $ git for-each-ref --format '%(objecttype) %(refname)' \
  | sed -n 's/^commit refs\/remotes\///p' \
  | while read -r type refname
do
  echo ref is "'$refname'"
done
ref is ''
ref is ''
ref is ''
ref is ''
...

madree explained this to me as: “the sed expression comes up with just freebsd/HEAD and freebsd/main for me and then read can’t possibly read two columns (type and refname). try omitting the “type” word and use just “while read -r refname” and see what you get”.

And so I did:

[dan@mydev:~/src/repos/ports] $ git for-each-ref --format '%(objecttype) %(refname)'   | sed -n 's/^commit refs\/remotes\///p'   | while read -r refname; do   echo ref is "'$refname'"; done
ref is 'origin/2014Q1'
ref is 'origin/2014Q2'
ref is 'origin/2014Q3'
ref is 'origin/2014Q4'
ref is 'origin/2015Q1'
ref is 'origin/2015Q2'
ref is 'origin/2015Q3'
ref is 'origin/2015Q4'
ref is 'origin/2016Q1'
ref is 'origin/2016Q2'
ref is 'origin/2016Q3'
ref is 'origin/2016Q4'
ref is 'origin/2017Q1'
ref is 'origin/2017Q2'
ref is 'origin/2017Q3'
ref is 'origin/2017Q4'
ref is 'origin/2018Q1'
ref is 'origin/2018Q2'
ref is 'origin/2018Q3'
ref is 'origin/2018Q4'
ref is 'origin/2019Q1'
ref is 'origin/2019Q2'
ref is 'origin/2019Q3'
ref is 'origin/2019Q4'
ref is 'origin/2020Q1'
ref is 'origin/2020Q2'
ref is 'origin/2020Q3'
ref is 'origin/2020Q4'
ref is 'origin/2021Q1'
ref is 'origin/2021Q2'
ref is 'origin/HEAD'
ref is 'origin/main'
[dan@mydev:~/src/repos/ports] $ 

This. This is more like it.

Getting the starting point

First, some background on the pseudo code.

At present, FreshPorts stores the ‘starting’ point, also known as the last commit, in a file: latest.ports.

Mathieu’s approach uses a tag in the repo itself. I was hesitant at first. If we lose the repo, we lose our starting points.

Line 18 from above has: if ! git tag -l freshports/$refname

That’s looking for the git tag. If we don’t find it, we know we have not processed this branch before. The typical use case here is a new quarterly branch has arrived and we want to process it. The first step of that processing is identifying the first commit on the branch.

Line 22 contains: git merge-base $NAME_OF_REMOTE/$NAME_OF_HEAD $refname)

NAME_OF_REMOTE and NAME_OF_HEAD confused me for a while. madree to the rescue.

[dan@mydev:~/src/repos/ports] $ git remote -v
origin	https://git.FreeBSD.org/ports.git (fetch)
origin	https://git.FreeBSD.org/ports.git (push)

[dan@mydev:~/src/repos/ports] $ git remote show
origin

There. NAME_OF_REMOTE is origin.

NAME_OF_HEAD is main. I can see that here:

[dan@mydev:~/src/repos/ports] $ git branch
* main
[dan@mydev:~/src/repos/ports] $ 

For this example, let’s use refname origin/2021Q2:

[dan@mydev:~/src/repos/ports] $ git merge-base origin/main origin/2021Q2
4e3cf0163c4a00d4dac41d6da43472d2fcab2f29

Looking at that commit:

author	Yuri Victorovich <yuri@FreeBSD.org>	2021-04-06 21:01:18 +0000
www/yt-dlp: Update 2021.03.24.1 -> 2021.04.03
PR:		254782
Submitted by:	daniel.engberg.lists@pyret.net

Is that really the first commit on the 2021Q2 branch?

I think it is not. But let’s continue with this exercise. Let’s see the first four commits.

[dan@mydev:~/src/repos/ports] $ git rev-list 4e3cf0163c4a00d4dac41d6da43472d2fcab2f29..origin/2021Q2 | head -4
5ceea227c504d2892d91c1aa8d8d81ff15b22fc3
3ce47d16f7eb5c00b470603c307fa52bb9ca920b
700466498a3c9c550882b91c9e9efec2ac533346
ce9f001bf8e1d5ef297e1a869cfb97f75f750c71
[dan@mydev:~/src/repos/ports] $ 

It is difficult for me to confirm whether or a commit is on a given branch. It’s not trivial.

Let’s try this for now, using the code from git_proc_commit

$ cd ~/src/freshports/git_proc_commit/git-to-freshports
$ ./git-to-freshports-xml.py --repo ports --path ~/src/repos/ports \
--commit-range 4e3cf0163c4a00d4dac41d6da43472d2fcab2f29..origin/2021Q2 \
--spooling ~/src/msgs/tmp --output ~/src/msgs/xml
$ ls ~/src/msgs/xml/ | wc -l
     590
$ grep -l 'Branch="2021Q2"' * | wc -l
     590

Those 590 files are now in the above repo as 2021Q2 examples. Everyone one of them contains Branch=”2021Q2″.

Future work

Future works means deleting those commits after fixing this issue:

Observer has noticed that commit 'eca26f0cd09d10ab7becf712579a1470768d966c' contains file /ports/2021Q2/sysutils/cbsd/distinfo as revision eca26f0cd09d10ab7becf712579a1470768d966c in repository ports
STARTING _CompileListOfPorts ................................
for a commit on 'branch': '2021Q2'
this commit is NOT ON head
FILE ==: Modify, /ports/2021Q2/sysutils/cbsd/Makefile, eca26f0cd09d10ab7becf712579a1470768d966c, ports, cbsd, Makefile, 4431078
YES, this file is in the ports tree
... but this file is not part of a physical category on disk!

Problem is, that is a valid physical category. On a branch. Adjustments are required.

In addition, that’s the wrong path. It should be /ports/branches/2021Q2 not /ports/2021Q2.

Fixed

The path thing is fixed. It was an issue on incoming commits, now repaired with:

 	my $NewRevision		= 0;
 	my $element;
 	my $element_id;
+	my $filename;
 	# This is where we add in the repo name to the path
-	my $filename     = $DB_Root_Prefix . '/' . $Updates{branch_for_files} . '/' . $FilePath;
+	# At one time, I think, $Updates{branch_for_files} was to be either head or branches/2021Q2 (or example).
+	# As of 2021.06.20, it is either head or 2021Q2 (no branches).
+	# I think the best thing to do is to check $Updates{branch_for_files} here and add in braches when required.
+	if ($Updates{branch_for_files} eq $FreshPorts::Constants::HEAD) {
+		$filename = $DB_Root_Prefix . '/' .          $Updates{branch_for_files} . '/' . $FilePath;
+	} else {
+		$filename = $DB_Root_Prefix . '/branches/' . $Updates{branch_for_files} . '/' . $FilePath;
+	}	

The 560 commits from 2021Q2 are now being imported.

This blog post has been updated throughout the day. It is now finished. Next, I’ll write up what’s next.

May 312021
 

Today I discovered a problem which required the reprocessing of about 400 commits.

How is that accomplished?

  1. delete the commits from the database
  2. tell the ingress daemon to start processing from commit x

Deleting

This works so long as all the commits have been process in order. The canonical method would query git to get a list of commits from A to B and then delete only those commits.

If you’ve been messing about with other commits and rerunning them, the method you see below may delete more than you anticipate.

The last successful commit was d9058fd29e8c3521cdece20a56a0ffdde4c1647f. This SQL deletes all commits which came in after that commit. This takes advantage of increasing values in the id field.

freshports.testgit=# begin; delete from commit_log where id > (select id from commit_log where svn_revision = 'd9058fd29e8c3521cdece20a56a0ffdde4c1647f');
BEGIN


DELETE 394
freshports.testgit=# 
freshports.testgit=# 
freshports.testgit=# commit;
COMMIT
freshports.testgit=# 

Reprocessing

This step involves resetting the starting point for commit processing.

[dan@testgit-ingress01:/var/db/ingress/repos] $ cat latest.ports 
3aa7a82534ec67d837a2e305480b45007f3d9b15
[dan@testgit-ingress01:/var/db/ingress/repos] $ echo 'echo d9058fd29e8c3521cdece20a56a0ffdde4c1647f > latest.ports' | sudo su -fm ingress
[dan@testgit-ingress01:/var/db/ingress/repos] cat latest.ports 
d9058fd29e8c3521cdece20a56a0ffdde4c1647f
[dan@testgit-ingress01:/var/db/ingress/repos] $ 

EDIT:

What the delete fails to recognize is it will delete all commits, src, doc, and ports.

Therefore, you have to rerun commits for src and doc, not just ports.

[dan@testgit-ingress01:/var/db/ingress/repos] echo 'echo 1f76329528e0a19a002ca0c174819bc4290fe04d > latest.doc' | sudo su -fm ingress
[dan@testgit-ingress01:/var/db/ingress/repos] echo 'echo 5e912f5fec025766521f535d1237330ede7f18e2 > latest.src' | sudo su -fm ingress