Over the past few weeks, I’ve made some progress towards supporting the branches for FreshPorts. Today, I started making the first code changes and I wanted to share that with you as I went along.
Finding the messages
FreshPorts processes commits by parsing the messages from the mailing list. I think it might be feasible to operate entirely off svn, and I recall discussing that with someone in the past. I was unable to find a blog post about it.
I started by looking at the mailing list archives. I found this message and selected it as the test candidate. However, that link doesn’t provide me with the raw email required for the FreshPorts queues. After a few minutes of panic and wonder over where to locate the raw text, I looked up a link from one of the FreshPorts entries. That led me to the archives on the docs host. From there, it didn’t take me long to find the same message. I used the Raw E-Mail link and grabbed the raw text I needed.
First code changes
This was the first code change. This allows the processing of messages from the new list.
$ svn di -x -ub process_mail.pl
Index: process_mail.pl
===================================================================
--- process_mail.pl (revision 4570)
+++ process_mail.pl (working copy)
@@ -58,7 +58,8 @@
if ($MessageId =~ /\@svn.freebsd.org/i || $MessageId =~ /\@svn.chruetertee.ch/i) {
if ($ListId =~ /SVN commit messages for the entire src tree/i ||
$ListId =~ /SVN commit messages for the entire doc trees/i ||
- $ListId =~ /SVN commit messages for the ports tree for head/i) {
+ $ListId =~ /SVN commit messages for the ports tree for head/i ||
+ $ListId =~ /SVN commit messages for all the branches of the ports tree/i) {
$found = 1;
# print "we should invoke the SVN scripts here\n";
eval "use process_svn_mail";
Running the email through this script gets me the XML for this message.
$ perl process_mail.pl pl < ~/tmp/msgs/branch01.txt
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE UPDATES PUBLIC "-//FreshPorts//DTD FreshPorts 2.0//EN" "http://www.freshports.org/docs/fp-updates.dtd">
<UPDATES Version="1.3.2.1">
<UPDATE>
<DATE Year="2012" Month="11" Day="8"></DATE>
<TIME Timezone="UTC" Hour="10" Second="1" Minute="18"></TIME>
<OS Id="FreeBSD" Repo="" Branch="UNKNOWN"></OS>
<LOG>MFH r307157 by gahr:
- fix build on amd64
Reported by: PH (via beat@)
Feature safe: yes</LOG>
<PEOPLE>
<UPDATER Handle="beat"></UPDATER>
</PEOPLE>
<MESSAGE EncodingLosses="false" Subject="svn commit: r307162 - branches/RELENG_9_1_0/games/spellathon" Id="201211081018.qA8AI1Gi047861@svn.freebsd.org">
<DATE Year="2012" Month="11" Day="8"></DATE>
<TIME Timezone="UTC" Hour="10" Second="1" Minute="18"></TIME>
<REPOSITORY></REPOSITORY>
<REVISION>307162</REVISION>
<TO Email="ports-committers@freebsd.org"></TO>
<TO Email="svn-ports-all@freebsd.org,"></TO>
</MESSAGE>
<FILES>
<FILE Revision="307162" Action="Modify" Path="branches/RELENG_9_1_0/games/spellathon/Makefile"></FILE>
</FILES>
</UPDATE>
</UPDATES>
I can see four potential problems here:
- Repo is empty
- Branch is empty
- REPOSITORY is empty
- Path may be an issue later, but looking at samples from HEAD, that looks correct
I made these changes and things looked much better.
$ svn di -x -ub process_svn_mail.pm
Index: process_svn_mail.pm
===================================================================
--- process_svn_mail.pm (revision 4570)
+++ process_svn_mail.pm (working copy)
@@ -455,6 +455,9 @@
elsif (m@\s+stable/(\d+)@) {
$branch = "RELENG_$1";
}
+ elsif (m@\s+branches/(RELENG[_\d]+)/@) {
+ $branch = "$1";
+ }
elsif (m@\s+vendor/@) {
$branch = "VENDOR";
}
@@ -481,7 +484,8 @@
my %KnownRepos = (
"SVN commit messages for the entire src tree" => $FreshPorts::Config::Repo_SRC,
"SVN commit messages for the entire doc trees" => $FreshPorts::Config::Repo_DOC,
- "SVN commit messages for the ports tree for head" => $FreshPorts::Config::Repo_PORTS
+ "SVN commit messages for the ports tree for head" => $FreshPorts::Config::Repo_PORTS,
+ "SVN commit messages for all the branches of the ports tree" => $FreshPorts::Config::Repo_PORTS
);
while (my ($ListId, $repo) = each %KnownRepos)
The first patch adds in support for the RELENG branches.
That second patch uses the same string you saw above in the previous patch. Yes, it would be better to have added that string to just one place. I welcome patches for that change.
With those changes, the XML now looks like this:
$ perl process_mail.pl < ~/tmp/msgs/branch01.txt
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE UPDATES PUBLIC "-//FreshPorts//DTD FreshPorts 2.0//EN" "http://www.freshports.org/docs/fp-updates.dtd">
<UPDATES Version="1.3.2.1">
<UPDATE>
<DATE Year="2012" Month="11" Day="8"></DATE>
<TIME Timezone="UTC" Hour="10" Second="1" Minute="18"></TIME>
<OS Id="FreeBSD" Repo="ports" Branch="RELENG_9_1_0"></OS>
<LOG>MFH r307157 by gahr:
- fix build on amd64
Reported by: PH (via beat@)
Feature safe: yes</LOG>
<PEOPLE>
<UPDATER Handle="beat"></UPDATER>
</PEOPLE>
<MESSAGE EncodingLosses="false" Subject="svn commit: r307162 - branches/RELENG_9_1_0/games/spellathon" Id="201211081018.qA8AI1Gi047861@svn.freebsd.org">
<DATE Year="2012" Month="11" Day="8"></DATE>
<TIME Timezone="UTC" Hour="10" Second="1" Minute="18"></TIME>
<REPOSITORY>ports</REPOSITORY>
<REVISION>307162</REVISION>
<TO Email="ports-committers@freebsd.org"></TO>
<TO Email="svn-ports-all@freebsd.org,"></TO>
</MESSAGE>
<FILES>
<FILE Revision="307162" Action="Modify" Path="branches/RELENG_9_1_0/games/spellathon/Makefile"></FILE>
</FILES>
</UPDATE>
</UPDATES>
That gives us good values for Repo, Branch, and REPOSITORY.
Loading the XML into the database
I saved the XML to a file and used that with the XML load script:
$ perl load_xml_into_db.pl ~/tmp/msgs/branch01.xml
about to process
Observer has noticed that processing has begun.
finished setting up the Parser
Processing file [/usr/home/dan/tmp/msgs/branch01.xml]...
dbname = freshports.org
parsing file now
*** start of all updates ***
--- start of an update ---
--- end of OS ---
OS is 'FreeBSD' : branch = 'RELENG_9_1_0;
sql = 'select SystemIDGet('FreeBSD')'
sql = 'select SystemBranchIDGet(1, 'RELENG_9_1_0')'
creating new Branch RELENG_9_1_0
OS is 'FreeBSD' (1) : branch = RELENG_9_1_0 (125)
found Committer= [beat]
***ports-committers@freebsd.org
found To = [ports-committers@freebsd.org]
...svn-ports-all@freebsd.org,
found To = [ports-committers@freebsd.org, svn-ports-all@freebsd.org,]
OS = [FreeBSD]
Branch = [RELENG_9_1_0]
Committer = [beat]
Date = [2012/11/08 10:18:01 UTC]
Log = [MFH r307157 by gahr:
- fix build on amd64
Reported by: PH (via beat@)
Feature safe: yes]
Repository = [ports]
Repository = [307162]
MessageId = [201211081018.qA8AI1Gi047861@svn.freebsd.org]
MessageDate = [2012/11/08 10:18:01 UTC]
MessageTo = [ports-committers@freebsd.org, svn-ports-all@freebsd.org,]
MessageSubject = [svn commit: r307162 - branches/RELENG_9_1_0/games/spellathon]
into handle_message_end, let's save that message now!
load_xml_into_db.pl::SaveUpdateToDB --- start
GetExistingMessageID => sql=select id from commit_log where message_id = '201211081018.qA8AI1Gi047861@svn.freebsd.org'
sql is insert into commit_log (id, message_id, message_date, message_subject, date_added, commit_date,
committer, description, system_id, svn_revision, repo_id, encoding_losses) values (
504798,
'201211081018.qA8AI1Gi047861@svn.freebsd.org',
'2012/11/08 10:18:01 UTC',
'svn commit: r307162 - branches/RELENG_9_1_0/games/spellathon',
'now()',
'2012/11/08 10:18:01 UTC',
'beat',
'MFH r307157 by gahr:
- fix build on amd64
Reported by: PH (via beat@)
Feature safe: yes',
1,
'307162',
(SELECT id FROM repo WHERE name = 'ports'),
false::boolean)
we have saved with id = '504798'
Observer has noticed that commit '201211081018.qA8AI1Gi047861@svn.freebsd.org' has been saved with a commit log id of 504798. Thank you.
load_xml_into_db.pl::SaveUpdateToDB --- finish
File = [Modify : /ports/branches/RELENG_9_1_0/games/spellathon/Makefile : 307162]
FileActionValid ==> M
sql = 'select Pathname_ID('/ports/branches/RELENG_9_1_0/games/spellathon/Makefile')'
sql = 'select Pathname_ID('/ports/branches/RELENG_9_1_0/games/spellathon')'
sql = 'select Pathname_ID('/ports/branches/RELENG_9_1_0/games/spellathon/Makefile')'
sql = 'insert into element_revision (element_id, revision_name) values (559873, '307162')'
saving commit_log_element
$FreshPorts::Constants::commit_log_seq='commit_log_id_seq'
$FreshPorts::Constants::ports_seq='ports_id_seq'
$FreshPorts::Constants::commit_log_elements_seq='commit_log_elements_id_seq'
commit_log_element::save..........
$FreshPorts::Constants::commit_log_seq='commit_log_id_seq'
$FreshPorts::Constants::ports_seq='ports_id_seq'
$FreshPorts::Constants::commit_log_elements_seq='commit_log_elements_id_seq'
getting id from 'commit_log_elements_id_seq'
sql is insert into commit_log_elements(id, commit_log_id, element_id, revision_name, change_type) values
(2377988, 504798, 559873, '307162', 'M')
sql = 'select ElementTagSet(125, 559873, '307162')'
pushing the following onto @Files
FileAction='Modify'
FilePath='/ports/branches/RELENG_9_1_0/games/spellathon/Makefile'
FileRevision='307162'
commit_log_element->{id}='2377988'
element_id='559873'
Observer has noticed that commit '201211081018.qA8AI1Gi047861@svn.freebsd.org' contains file /ports/branches/RELENG_9_1_0/games/spellathon/Makefile as revision 307162 in repository ports
STARTING _CompileListOfPorts ................................
FILE ==: Modify, /ports/branches/RELENG_9_1_0/games/spellathon/Makefile, 307162, ports, RELENG_9_1_0, games, spellathon/Makefile, 2377988
YES, this file is in the ports tree
checking for category='RELENG_9_1_0'
sql = "select * from categories where name = 'RELENG_9_1_0'"
NOT FOUND
creating new category RELENG_9_1_0
creating new category RELENG_9_1_0
DESTDIR=/usr/local/FreshPorts/PORTS-SVN/RELENG_9_1_0
SRCDIR =ports/RELENG_9_1_0
FILE =Makefile
before 'ports/RELENG_9_1_0'
after 'RELENG_9_1_0'
about to fetch = 'sh /usr/local/FreshPorts/scripts/svn-up-file.sh SVNDIR SVNITEM REVISION /usr/local/FreshPorts/PORTS-SVN/RELENG_9_1_0 RELENG_9_1_0 Makefile HEAD '''
fetch result = 256
that fetch failed. What do to?
num of params = 8
error invoking script /usr/local/FreshPorts/scripts/svn-up-file.sh : usage /usr/local/FreshPorts/scripts/svn-up-file.sh SVNDIR SVNITEM REVISION (e.g. /usr/local/FreshPorts/scripts/svn-up-file.sh /usr/ports sysutils/bacula-server 1234)
fetch failed, sleeping...
Could not fetch file for '/usr/local/FreshPorts/PORTS-SVN/RELENG_9_1_0' 'ports/RELENG_9_1_0' 'Makefile'. Error code = 1
checking for port='RELENG_9_1_0/games'
* * * not found in existing cache. we'll have to load/create that port!
sql = 'select Pathname_ID('/ports/branches/RELENG_9_1_0/games')'
sql = '
select ports.*,
categories.name as category,
element.name as name,
element_pathname(element.id, FALSE) as element_pathname
from ports, categories, element
where ports.element_id = 559871
and ports.category_id = categories.id
and ports.element_id = element.id'
port not retrieved with /ports/branches/RELENG_9_1_0/games. This must be a new port
SETTING CATEGORY = 125
ENDING _CompileListOfPorts ................................
into SaveChangesToPortsTree()
port = RELENG_9_1_0/games
category_id='125', needs_refresh='7'
into FreshPorts::Port::save
sql is insert into ports (id, element_id, category_id, last_commit_id) values (
34149,
559871,
125, 504798)
about to delete port_dependencies for id 34149
sql is DELETE FROM port_dependencies WHERE port_id = 34149
no depends found
depends with this: ''
no depends to look for; returning
no depends found
depends with this: ''
no depends to look for; returning
no depends found
depends with this: ''
no depends to look for; returning
just created that port. Now loading it back in....
sql is insert into commit_log_ports
(commit_log_id, port_id, needs_refresh, port_version, port_revision) values
(504798, 34149, 7,
NULL, NULL)
size of %CommitLogPorts for RELENG_9_1_0/games is '1'
That message is all done under Commit ID = '504798'
the size of @Files is 1
FILE ==: Modify, /ports/branches/RELENG_9_1_0/games/spellathon/Makefile, 307162, , ports, branches, RELENG_9_1_0/games/spellathon/Makefile, 2377988
# # # # Deleting deleted ports # # # #
# # # # Finished deleting deleted ports # # # #
# # # # Resurrecting deleted ports # # # #
# # # # Finished resurrecting deleted ports # # # #
into _RecordPortsAndElements
That message is all done under Commit ID = '504798'
the size of @Files is 1
checking file '/ports/branches/RELENG_9_1_0/games/spellathon/Makefile' : element_id = '559873'
checking port ports/branches/RELENG_9_1_0/games
YES! that was a match!
saving to commit_log_ports_elements
RELENG_9_1_0/games
sql is insert into commit_log_ports_elements
(commit_log_id, element_id) values
(504798, 559871)
done _RecordPortsAndElements
--- end of this update ---
this commit is from the 'ports' repository.
oh, the script goes to fetch...
fetching all files from this commit.
after ''
about to svn up = 'sh /usr/local/FreshPorts/scripts/svn-up-file.sh /usr/local/FreshPorts/PORTS-SVN '' 307162'
It was at this point that I realized FreshPorts really needs to maintain multiple port trees. One for HEAD, one for each branch. That is, I don’t want to be svn up’ing from HEAD to RELENG and back to HEAD again…
This will need some more thought. I’m quite sure it can be done, but I just need to code it.
Of more importance, the code is looking at the RELENG value as a category. I’ve got to fix that. If you look above, it just created a RELENG_9_1_0 category. I’ll work on that next.
My next step is pretty simple: get that branch.
svn co -r 307162 svn://svn.freebsd.org/ports/branches/RELENG_9_1_0 PORTS-RELENG_9_1_0
By the way, if you want to see that failed commit as loaded into the database, it’s here.
