Fetch, Patch, & Extract dependencies and Uses

FreshPorts is not magic. It grabs the stuff it needs by running make -V with a number of parameters. New fields are added from time to time, usually at the prompting of a user. Recently, the following fields were added to the dev website:

FETCH_DEPENDS
PATCH_DEPENDS
EXTRACT_DEPENDS
USES

More information on these fields can be found in the Dependencies section of the FreeBSD Porter’s Handbook.

Other recent changes, already in production include:

Find issues related to this port
Report an issue related to this port
The format used to display dependencies has been changed to show the file/pkg in question (e.g. python2) and the port/pkg it is supplied from (e.g. lang/python2)

The rest of this post will outline some of the changes made to dev, and a CATEGORIES bug discovered during the database population phase of these changes.

Broken CATEGORIES

While testing the above changes, I noticed 24 ports had a NULL value for the categories column of the ports table:

freshports.org=# select name, category, element_pathname(element_id) from ports_active where categories is null;
                 name                  |  category  |                            element_pathname                            
---------------------------------------+------------+------------------------------------------------------------------------
 rubygem-inflecto                      | devel      | /ports/head/devel/rubygem-inflecto
 py-wcwidth                            | devel      | /ports/head/devel/py-wcwidth
 py-Products.GenericSetup              | devel      | /ports/branches/RELENG_9_1_0/devel/py-Products.GenericSetup
 py-eggtestinfo                        | devel      | /ports/branches/RELENG_9_1_0/devel/py-eggtestinfo
 rubygem-request_store                 | devel      | /ports/head/devel/rubygem-request_store
 calligra                              | editors    | /ports/branches/RELENG_9_1_0/editors/calligra
 xzx                                   | emulators  | /ports/branches/RELENG_9_1_0/emulators/xzx
 gexiv2                                | graphics   | /ports/branches/RELENG_9_1_0/graphics/gexiv2
 gource                                | graphics   | /ports/branches/RELENG_9_1_0/graphics/gource
 squirrelmail                          | mail       | /ports/branches/RELENG_9_1_0/mail/squirrelmail
 devede                                | multimedia | /ports/branches/RELENG_9_1_0/multimedia/devede
 jstrack                               | science    | /ports/branches/RELENG_9_1_0/science/jstrack
 bro                                   | security   | /ports/branches/RELENG_9_1_0/security/bro
 php53-openssl                         | security   | /ports/branches/RELENG_9_2_0/security/php53-openssl
 createrepo                            | sysutils   | /ports/branches/RELENG_9_1_0/sysutils/createrepo
 p5-Corona                             | www        | /ports/branches/RELENG_9_1_0/www/p5-Corona
 p5-POE-Component-Server-PSGI          | www        | /ports/branches/RELENG_9_1_0/www/p5-POE-Component-Server-PSGI
 p5-Plack-Handler-AnyEvent-ReverseHTTP | www        | /ports/branches/RELENG_9_1_0/www/p5-Plack-Handler-AnyEvent-ReverseHTTP
 py-requests-toolbelt                  | www        | /ports/head/www/py-requests-toolbelt
 py-rhodecode                          | www        | /ports/branches/RELENG_9_1_0/www/py-rhodecode
 py-satchmo                            | www        | /ports/branches/RELENG_9_1_0/www/py-satchmo
 razorback-dispatcher                  | security   | /ports/branches/RELENG_9_1_0/security/razorback-dispatcher
 rtorrent-devel                        | net-p2p    | /ports/branches/RELENG_9_1_0/net-p2p/rtorrent-devel
 php53                                 | lang       | /ports/branches/RELENG_9_2_0/lang/php53
(24 rows)

freshports.org=#

These errors came to light when running a script to populate the newly created, and initially empty, fields: fetch_depends, extract_depends, patch_depends, and uses. Those errors were:

Use of uninitialized value in split at ..//port.pm line 861.
Use of uninitialized value in concatenation (.) or string at ..//port.pm line 874.
Use of uninitialized value in substitution (s///) at ..//port.pm line 824.

It took me a while to track it down using the log file created during the processing of each commit. This bug was introduced during the coding and is not present in production. This sounded much more interesting while I was writing up, but I didn’t know then that the bug was so recent and isolated.

You can see the issue here:


--- port.pm     (revision 4692)
+++ port.pm     (working copy)
@@ -821,6 +821,7 @@
                # convert all whitespace to a single space
                # This arose from 200609130717.k8D7HpNc057638@repoman.freebsd.org
                #
+               $this->{categories}                     = $categories;
                $this->{categories}                     =~ s/\s+/ /g;
 
                $result = $this->_Validate();

Being branch aware

While coding these changes, I realized that the code had to be much more branch aware than previously designed. The code for processing a commit is aware of what branch the commit is done, and updates only those ports on that branch. However, the code for refreshing the database, when new fields are added, was not.

Getting the Branch from the path name

When processing the ports for refresh, a process not often done, the SQL has been altered to return the full pathname of the port. This is used to determine the branch we are upgrading. Two new functions were created: one to grab the branch from the path, and the other to set the branch in the database so it knows what to do.

Index: branches.pm
===================================================================
--- branches.pm (revision 4692)
+++ branches.pm (working copy)
@@ -109,4 +109,52 @@
   }
 }
 
+sub GetBranchFromPathName($)
+{
+  my $pathname = shift();
+
+  # convert /ports/branches/2014Q1/archivers/hs-tar to 2014Q1
+  # convert /ports/head/archivers/hs-tar            to head
+  # convert /ports/head/accessibility/accerciser    to head
+
+  # the variable names used here assume HEAD  
+  #
+  my ($emptyLeadingSlash, $subtree, $branch, $category, $port) = split/\//,$pathname, 5;
+
+  # example result based on above
+  #        undefined            'ports'    'head'  'archivers'  'hs-tar'
+
+  print "GetBranchFromPathName finds: '$subtree', '$category', '$port'\n";
+  if ($subtree ne $FreshPorts::Constants::PORTS) {
+    die("Unrecognized structure for pathname('$pathname'): $emptyLeadingSlash, $subtree, $category, $port");
+  }
+
+  if ($branch eq $FreshPorts::Constants::HEAD)
+  {
+    return $branch;
+  } else {
+    if ($branch eq 'branches')
+    {
+      return $category;
+    } else {
+      die("Unable to determine branch for '$pathname'\n");
+    }
+  }
+
+  die("Faulty code logic.  We should never get here in GetBranchFromPathName\n");
+}
+
+sub SetBranchInDB($$) {
+  my $dbh    = shift();
+  my $branch = shift();
+
+  $sql = 'select freshports_branch_set(' . $dbh->quote($branch) . ')';
+  $sth = $dbh->prepare($sql);
+  if (!$sth->execute())  {
+    FreshPorts::Utilities::ReportError('warning', "SetBranchInDB: Could not set branch:" .  $dbh->errstr, 1);
+  }
+  
+  $sth->finish();
+}
+
 1;

Lost time

I had trouble figuring the Branch problem. The update process kept adding dependencies to ports on the wrong branch, resulting in duplicate key errors. I incorrectly assumed the problem was related to adding duplicate and wrote some code to keep track of what had been added and not add duplicates. The problem was actually related to adding dependencies to ports not under consideration (i.e. the right port, but in the wrong branch). I’m re-running the update/population script now, with the duplicate-avoidance code removed.