In the last post, I found that many commits were to the master branch when they should have been on the quarterly branch. Now I think I see why.
See this XML:
<OS Repo="ports-quarterly" Id="FreeBSD" Branch="master"/>
If it’s quarterly, it should name the branch. Case in point: 2020Q4.
I went to https://lists.freebsd.org/pipermail/svn-ports-branches/2020-November/thread.html to look for known quarterly commits.
Hmm, first, let’s find known commits at https://github.com/freebsd/freebsd-ports/tree/branches/2020Q4 – when I looked, the latest commit was https://github.com/freebsd/freebsd-ports/commit/46433baae934d92698422495b72f811839caa1a9
MFH: r555565 security/wolfssl: fix build on big-endian Merge upstream patch to fix build on big-endian architectures. Also unmark mips and mips64 as broken, now builds fine. Approved by: portmgr (fix build blanket)
The commit just before that is: https://github.com/freebsd/freebsd-ports/commit/e79616836f4e962d370f4364760d85a5e8460a65
How do did I find out? I looked at https://github.com/freebsd/freebsd-ports/commits/branches/2020Q4
When FreshPorts processes a commit, it needs a working copy of the repo as it looked at that commit.
I am trying to figure out how to do that when the commit is on a branch.
To get a copy of the branch, I do:
$ git checkout branches/2020Q4 $ git branch branches/2020Q3 * branches/2020Q4 master
Next, I want the tree as it existed at commit 46433baae934d92698422495b72f811839caa1a9
i.e. https://github.com/freebsd/freebsd-ports/commit/46433baae934d92698422495b72f811839caa1a9
My first attempt is
$ git checkout 46433baae934d92698422495b72f811839caa1a9 Note: switching to '46433baae934d92698422495b72f811839caa1a9'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -cOr undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 46433baae934 MFH: r555565
That MFH: r555565 message indicates that I am at the right commit.
Now, if I process that commit:
echo /usr/local/libexec/freshports/git-to-freshports-xml.py --repo ports-quarterly --path \ /usr/home/dan/src/freebsd/freebsd-ports-quarterly --single-commit \ 46433baae934d92698422495b72f811839caa1a9 --spooling /var/db/ingress/message-queues/spooling --output \ /tmp -v | sudo su -fm ingress
The important point from the above: /usr/home/dan/src/freebsd/freebsd-ports-quarterly
This is a checkout of the quarterly ports branch.
In the file I get:
$ cat /tmp/2020.11.17.16.07.01.000000.46433baae934d92698422495b72f811839caa1a9.xml
<?xml version='1.0' encoding='UTF-8'?>
<UPDATES Version="1.4.0.0">
<UPDATE>
<DATE Year="2020" Month="11" Day="17"/>
<TIME Timezone="UTC" Hour="16" Minute="7" Second="1"/>
<OS Repo="ports-quarterly" Id="FreeBSD" Branch="branches/2020Q4"/>
<LOG>MFH: r555565
security/wolfssl: fix build on big-endian
Merge upstream patch to fix build on big-endian architectures.
Also unmark mips and mips64 as broken, now builds fine.
Approved by: portmgr (fix build blanket)</LOG>
<PEOPLE>
<UPDATER Handle="pkubaj <pkubaj@FreeBSD.org>"/>
</PEOPLE>
<COMMIT Hash="46433baae934d92698422495b72f811839caa1a9" HashShort="46433ba" Subject="MFH: r555565" EncodingLoses="false" Repository="ports-quarterly"/>
<FILES>
<FILE Action="Modify" Path="security/wolfssl/Makefile"/>
<FILE Action="Modify" Path="security/wolfssl/distinfo"/>
</FILES>
</UPDATE>
</UPDATES>
$
OK, that has the expected branch value for Branch, but those files are wrong. Looking on dev, I find this for the same commit:
<FILE Action="Modify" Path="branches/2020Q4/security/wolfssl/Makefile" Revision="555566"></FILE>
<FILE Revision="555566" Path="branches/2020Q4/security/wolfssl/distinfo" Action="Modify"></FILE>
<FILE Revision="555566" Action="Modify" Path="branches/2020Q4/"></FILE>
Notice they all start with branches/2020Q4.
A prefix is missing. We could handle this on the XML processing side.
Lets’ see.
We have this in the XML: security/wolfssl/Makefile
We have this in debugging: File = [Modify : /ports/2020Q4/security/wolfssl/Makefile : 46433baae934d92698422495b72f811839caa1a9]
That output is produced by this code in xml_munge_git.pm:
# This is where we add in the repo name to the path
my $filename = $DB_Root_Prefix . '/' . $Updates{branch} . '/' . $FilePath;
my $revisionname = $FileRevision;
my $commit_log_element;
print "File = [$FileAction : $filename";
I took about an hour to figure this out. And made code changes:
Index: xml_munge_git.pm
===================================================================
--- xml_munge_git.pm (revision 5489)
+++ xml_munge_git.pm (working copy)
@@ -214,8 +214,11 @@
$p->register(">UPDATES>UPDATE>OS:Id", "attr" => \$Updates{os});
#
- # for git, let's put branch in branch-git
- # will will populate $Updates{branch} with the converted value. e.g. master -> head
+ # EDIT 2020-11-17 - Updates{branch_git} is the branch name supplied by git. e.g. master, branches/20202Q4
+ # EDIT 2020-11-17 - removing all references to $Updates{branch}
+ #
+ # for git, let's put branch in branch_git
+ # will will populate $Updates{} with the converted value. e.g. master -> head
# and branches/2020Q3 -> 2020Q3
#
$p->register(">UPDATES>UPDATE>OS:Branch", "attr" => \$Updates{branch_git});
@@ -272,7 +275,7 @@
# XXX delete
# my $branch_name = FreshPorts::Branches::stripBranchesToGetBranchName($BranchName);
-# $Update{branch_name} = $branch_name;
+# $Updates{branch_name} = $branch_name;
print "OS is '$Updates{os}' : branch = '$Updates{branch_git}' for git\n";
@@ -279,16 +282,42 @@
# When we moved from subversion to git, we needed to convert branch from
# master to head, because everything we need here is based on head.
#
- # $Updates{branch} : for database related actions (finding a port) e.g. head or 2020Q3
- # $Updates{branch_git} : for repository related actions (git checkout)
+ # $Updates{branch_for_files} : for database related actions (finding a port) e.g. head or 2020Q3
+ # $Updates{branch_git} : for repository related actions (git checkout)
+ #
+ # In the system_branch.branch_name column, we have values such as 2020Q4 and
+ # the prefix 'branches' is not included.
+ #
+ # But for files, the prefix is included:
+ # freshports.dev=# select * from element_pathname where pathname like '/ports/branches/2019Q3/%' limit 5;
+ # element_id | pathname
+ # ------------+-------------------------------------------------
+ # 960349 | /ports/branches/2019Q3/MOVED
+ # 954142 | /ports/branches/2019Q3/Mk
+ # 954143 | /ports/branches/2019Q3/Mk/Scripts
+ # 954144 | /ports/branches/2019Q3/Mk/Scripts/do-depends.sh
+ # 956066 | /ports/branches/2019Q3/Mk/Uses
+ # (5 rows)
+ # freshports.dev=#
+ #
+ #
+ # So we have the following values:
+ #
+ # $Updates{branch_git} - value supplied in XML
+ # $Updates{branch_database_name} - for use in system_branch.branch_name
+ # $Updates{branch_for_files} - for use in filenames
+ #
- $Updates{branch} = ConvertGitBranch($Updates{branch_git});
+ # this converts master to head, and leaves everything else unchanged
+ #
+ $Updates{branch_for_files} = ConvertGitBranchNameToFreshPortsName($Updates{branch_git});
- print "after converting '\$Updates{branch_git}' we have '$Updates{branch_git}'\n";
+ print "after converting '\$Updates{branch_git}' we have '$Updates{branch_for_files}'\n";
print "next we need to strip any leading 'branches/' prefix\n";
- $Updates{branch} = FreshPorts::Branches::stripBranchesToGetBranchName($Updates{branch});
- print "OS is '$Updates{os}' : branch = '$Updates{branch}'\n";
+ $Updates{branch_database_name} = FreshPorts::Branches::stripBranchesToGetBranchName($Updates{branch_for_files});
print "OS is '$Updates{os}' : branch = '$Updates{branch_git}' for git\n";
+ print "OS is '$Updates{os}' : branch = '$Updates{branch_for_files}' for git\n";
+ print "OS is '$Updates{os}' : branch = '$Updates{branch_database_name}' for database names\n";
# We know what branch this message is updating. Let's grab the IDs we will need.
$SystemID = SystemIDGet($Updates{os}, $self->{dbh});
@@ -297,12 +326,12 @@
FreshPorts::Utilities::ReportError('warning', "No SystemID found for OS = '$Updates{os}'", 1)
}
- if ($Updates{branch} ne '') {
+ if ($Updates{branch_database_name} ne '') {
# we invoke GetBranchFromPathName to convert branches/2020Q3 to 2020Q3
- $SystemBranchID = SystemBranchIDGetOrCreate($SystemID, $Updates{branch}, $self->{dbh});
+ $SystemBranchID = SystemBranchIDGetOrCreate($SystemID, $Updates{branch_database_name}, $self->{dbh});
if (!defined($SystemBranchID)) {
$! = 4;
- FreshPorts::Utilities::ReportError('warning', "No SystemBranchID found for OS = '$Updates{branch}'", 1);
+ FreshPorts::Utilities::ReportError('warning', "No SystemBranchID found for OS = '$Updates{branch_database_name}'", 1);
} else {
$Updates{branch_id} = $SystemBranchID;
}
@@ -312,7 +341,7 @@
die "Branch was empty. Probably imported sources. Ignoring message $inputfile\n";
}
- print "OS is '$Updates{os}' ($SystemID) : branch = $Updates{branch} ($SystemBranchID)\n";
+ print "OS is '$Updates{os}' ($SystemID) : branch = $Updates{branch_database_name} ($SystemBranchID)\n";
}
@@ -338,7 +367,7 @@
FreshPorts::Utilities::ReportError('Err', "No files found in commit '$Updates{commit_hash}'. Has someone done a cvs import instead of addport?", 0)
}
- %CommitLogPorts = FreshPorts::VerifyPort::SaveChangesToPortsTree($Updates{branch}, commit_log_id(), \@Files, $self->{dbh});
+ %CommitLogPorts = FreshPorts::VerifyPort::SaveChangesToPortsTree($Updates{branch_database_name}, commit_log_id(), \@Files, $self->{dbh});
#
# commit what we have now, and that starts a new transaction.
@@ -429,7 +458,9 @@
# we don't clear these values until the end of the update
undef $Updates{os};
- undef $Updates{branch};
+ undef $Updates{branch_git};
+ undef $Updates{branch_database_name};
+ undef $Updates{branch_for_files};
undef $Updates{committerAll};
undef $Updates{dateyear};
undef $Updates{datemonth};
@@ -472,7 +503,7 @@
return $ValidFileActions{$FileAction};
}
-sub ConvertGitBranch($) {
+sub ConvertGitBranchNameToFreshPortsName($) {
my $GitBranch = shift;
#
@@ -610,7 +641,7 @@
my $element;
my $element_id;
# This is where we add in the repo name to the path
- my $filename = $DB_Root_Prefix . '/' . $Updates{branch} . '/' . $FilePath;
+ my $filename = $DB_Root_Prefix . '/' . $Updates{branch_for_files} . '/' . $FilePath;
my $revisionname = $FileRevision;
my $commit_log_element;
@@ -808,11 +839,12 @@
# The criteria for that is the subject must start with
# "cvs commit: ports/".
- print "OS = [$Updates{os}]\n";
- print "Branch git = [$Updates{branch_git}]\n";
- print "Branch = [$Updates{branch}]\n";
- print "Committer = [$Updates{committerAll}]\n";
- print "Date = [" . sprintf "%04u/%02u/%02u %02u:%02u:%02u %s", $Updates{dateyear}, $Updates{datemonth}, $Updates{dateday}, $Updates{timehour}, $Updates{timeminute}, $Updates{timesecond}, $Updates{timezone} . "]\n";
+ print "OS = [$Updates{os}]\n";
+ print "Branch git = [$Updates{branch_git}]\n";
+ print "branch_database_name = [$Updates{branch_database_name}]\n";
+ print "branch_for_files = [$Updates{branch_for_files}]\n";
+ print "Committer = [$Updates{committerAll}]\n";
+ print "Date = [" . sprintf "%04u/%02u/%02u %02u:%02u:%02u %s", $Updates{dateyear}, $Updates{datemonth}, $Updates{dateday}, $Updates{timehour}, $Updates{timeminute}, $Updates{timesecond}, $Updates{timezone} . "]\n";
if (defined($Updates{repository})) {
print "Repository = [$Updates{repository}]\n";
} else {
@@ -835,7 +867,7 @@
# First thing we must do, is tell the database what Branch to use...
# XXX why does this not use branches::SetBranchInDB() ?
- my $sql = 'select freshports_branch_set(' . $self->{dbh}->quote($Updates{branch}) . ')';
+ my $sql = 'select freshports_branch_set(' . $self->{dbh}->quote($Updates{branch_database_name}) . ')';
my $sth = $self->{dbh}->prepare($sql);
if (!$sth->execute()) {
FreshPorts::Utilities::ReportError('warning', "Could not set branch", 1);
[dan@devgit-ingress01:~/modules] $
But wait, there’s more!
There were more things wrong too. I was going git checkout master on branches, which meant all the branch commits were being processed as if they were on master. Bad. :/
Some of that came out in git commit processing – how is it done? which was published after I started this blog post, but before I finished it.
