FreshPorts is not magic. It grabs the stuff it needs by running make -V with a number of parameters. New fields are added from time to time, usually at the prompting of a user. Recently, the following fields were added to the dev website:
- FETCH_DEPENDS
- PATCH_DEPENDS
- EXTRACT_DEPENDS
- USES
More information on these fields can be found in the Dependencies section of the FreeBSD Porter’s Handbook.
Other recent changes, already in production include:
- Find issues related to this port
- Report an issue related to this port
- The format used to display dependencies has been changed to show the file/pkg in question (e.g. python2) and the port/pkg it is supplied from (e.g. lang/python2)
The rest of this post will outline some of the changes made to dev, and a CATEGORIES bug discovered during the database population phase of these changes.
Broken CATEGORIES
While testing the above changes, I noticed 24 ports had a NULL value for the categories column of the ports table:
freshports.org=# select name, category, element_pathname(element_id) from ports_active where categories is null; name | category | element_pathname ---------------------------------------+------------+------------------------------------------------------------------------ rubygem-inflecto | devel | /ports/head/devel/rubygem-inflecto py-wcwidth | devel | /ports/head/devel/py-wcwidth py-Products.GenericSetup | devel | /ports/branches/RELENG_9_1_0/devel/py-Products.GenericSetup py-eggtestinfo | devel | /ports/branches/RELENG_9_1_0/devel/py-eggtestinfo rubygem-request_store | devel | /ports/head/devel/rubygem-request_store calligra | editors | /ports/branches/RELENG_9_1_0/editors/calligra xzx | emulators | /ports/branches/RELENG_9_1_0/emulators/xzx gexiv2 | graphics | /ports/branches/RELENG_9_1_0/graphics/gexiv2 gource | graphics | /ports/branches/RELENG_9_1_0/graphics/gource squirrelmail | mail | /ports/branches/RELENG_9_1_0/mail/squirrelmail devede | multimedia | /ports/branches/RELENG_9_1_0/multimedia/devede jstrack | science | /ports/branches/RELENG_9_1_0/science/jstrack bro | security | /ports/branches/RELENG_9_1_0/security/bro php53-openssl | security | /ports/branches/RELENG_9_2_0/security/php53-openssl createrepo | sysutils | /ports/branches/RELENG_9_1_0/sysutils/createrepo p5-Corona | www | /ports/branches/RELENG_9_1_0/www/p5-Corona p5-POE-Component-Server-PSGI | www | /ports/branches/RELENG_9_1_0/www/p5-POE-Component-Server-PSGI p5-Plack-Handler-AnyEvent-ReverseHTTP | www | /ports/branches/RELENG_9_1_0/www/p5-Plack-Handler-AnyEvent-ReverseHTTP py-requests-toolbelt | www | /ports/head/www/py-requests-toolbelt py-rhodecode | www | /ports/branches/RELENG_9_1_0/www/py-rhodecode py-satchmo | www | /ports/branches/RELENG_9_1_0/www/py-satchmo razorback-dispatcher | security | /ports/branches/RELENG_9_1_0/security/razorback-dispatcher rtorrent-devel | net-p2p | /ports/branches/RELENG_9_1_0/net-p2p/rtorrent-devel php53 | lang | /ports/branches/RELENG_9_2_0/lang/php53 (24 rows) freshports.org=#
These errors came to light when running a script to populate the newly created, and initially empty, fields: fetch_depends, extract_depends, patch_depends, and uses. Those errors were:
Use of uninitialized value in split at ..//port.pm line 861. Use of uninitialized value in concatenation (.) or string at ..//port.pm line 874. Use of uninitialized value in substitution (s///) at ..//port.pm line 824.
It took me a while to track it down using the log file created during the processing of each commit. This bug was introduced during the coding and is not present in production. This sounded much more interesting while I was writing up, but I didn’t know then that the bug was so recent and isolated.
You can see the issue here:
--- port.pm (revision 4692) +++ port.pm (working copy) @@ -821,6 +821,7 @@ # convert all whitespace to a single space # This arose from 200609130717.k8D7HpNc057638@repoman.freebsd.org # + $this->{categories} = $categories; $this->{categories} =~ s/\s+/ /g; $result = $this->_Validate();
Being branch aware
While coding these changes, I realized that the code had to be much more branch aware than previously designed. The code for processing a commit is aware of what branch the commit is done, and updates only those ports on that branch. However, the code for refreshing the database, when new fields are added, was not.
Getting the Branch from the path name
When processing the ports for refresh, a process not often done, the SQL has been altered to return the full pathname of the port. This is used to determine the branch we are upgrading. Two new functions were created: one to grab the branch from the path, and the other to set the branch in the database so it knows what to do.
Index: branches.pm =================================================================== --- branches.pm (revision 4692) +++ branches.pm (working copy) @@ -109,4 +109,52 @@ } } +sub GetBranchFromPathName($) +{ + my $pathname = shift(); + + # convert /ports/branches/2014Q1/archivers/hs-tar to 2014Q1 + # convert /ports/head/archivers/hs-tar to head + # convert /ports/head/accessibility/accerciser to head + + # the variable names used here assume HEAD + # + my ($emptyLeadingSlash, $subtree, $branch, $category, $port) = split/\//,$pathname, 5; + + # example result based on above + # undefined 'ports' 'head' 'archivers' 'hs-tar' + + print "GetBranchFromPathName finds: '$subtree', '$category', '$port'\n"; + if ($subtree ne $FreshPorts::Constants::PORTS) { + die("Unrecognized structure for pathname('$pathname'): $emptyLeadingSlash, $subtree, $category, $port"); + } + + if ($branch eq $FreshPorts::Constants::HEAD) + { + return $branch; + } else { + if ($branch eq 'branches') + { + return $category; + } else { + die("Unable to determine branch for '$pathname'\n"); + } + } + + die("Faulty code logic. We should never get here in GetBranchFromPathName\n"); +} + +sub SetBranchInDB($$) { + my $dbh = shift(); + my $branch = shift(); + + $sql = 'select freshports_branch_set(' . $dbh->quote($branch) . ')'; + $sth = $dbh->prepare($sql); + if (!$sth->execute()) { + FreshPorts::Utilities::ReportError('warning', "SetBranchInDB: Could not set branch:" . $dbh->errstr, 1); + } + + $sth->finish(); +} + 1;
Lost time
I had trouble figuring the Branch problem. The update process kept adding dependencies to ports on the wrong branch, resulting in duplicate key errors. I incorrectly assumed the problem was related to adding duplicate and wrote some code to keep track of what had been added and not add duplicates. The problem was actually related to adding dependencies to ports not under consideration (i.e. the right port, but in the wrong branch). I’m re-running the update/population script now, with the duplicate-avoidance code removed.
Beta testing
This code will run on dev for a while before moving to production. Suggestions & comments always welcome.
This is now in production.