It takes a long time for FreshPorts to process a new vuln.xml file. A very long time. Why? Because it process all entries in the file. Every time.

Anton Berezin proposed an idea. I paraphrase, but in brief, he said:

Figuring out what changed is as easy as hashing an entry and storing it in a db

That’s a very good start. I’ve checked the code, and the sticking point is going to be is table:

freshports.org=# \d ports_vulnerable
    Table "public.ports_vulnerable"
 Column  |  Type   |     Modifiers
---------+---------+--------------------
 port_id | integer | not null
 current | integer | not null default 1
 past    | integer | not null default 0
Indexes:
    "ports_vulnerable_pkey" PRIMARY KEY, btree (port_id)
Foreign-key constraints:
    "$1" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE RESTRICT ON DELETE CASCADE
Triggers:
    ports_vulnerable_delete_clear_cache AFTER DELETE ON ports_vulnerable FOR EACH ROW EXECUTE PROCEDURE ports_vulnerable_delete_clear_cache()

freshports.org=#

This table is used to decide whether or not to display a grey or black skull at the top of each port page. This table is populated with two different queries. The first, figures out the number of vulnerabilities, current and past, that affect a given port:

  SELECT count(distinct vuxml_id)
    INTO l_VulnCount
    FROM commit_log_ports_vuxml
   WHERE port_id = p_PortID;

The second, determines the number of vulnerabilities that currently affect the port:

    SELECT count(distinct vuxml_id)
      INTO l_VulnCurrent
      FROM commit_log_ports_vuxml CLPV, commit_log_ports CLP, ports P, ports_vulnerable PV
     WHERE CLP.commit_log_id = CLPV.commit_log_id
       AND CLPV.port_id      = CLP.port_id
       AND P.id              = CLP.port_id
       AND P.version         = CLP.port_version
       AND P.revision        = CLP.port_revision
       AND CLP.port_epoch    = P.portepoch
       AND PV.port_id        = P.id
       AND P.id              = p_PortID
     GROUP BY P.id;

NOTE: I am not sure of the reason for referencing ports_vulnerable here… My first thought is it is unnecessary. I’m not sure how it is to reference a table that is just about to get updated.

The difference between the two values, if any, is the number of past vulnerabilities.

The problem is an edge case where a vuln entry affects a port, then does not affect a port. For example, consider that a new vuln entry is created and mistakenly refers to PortA instead of PortB. At present, we wipe all entries from the ports_vulnerable table each time we process a new vuln.xml file. Thus, we start from nil each time. If we are now processing just one vuln at a time, we may have a lingering entry in ports_vulnerable which should be deleted.

How do we detect this situation?

Well, when we process the new vuln, we will delete the old one first, from the vuxml table. When this happens, various triggers are set off to delete related records. We can harness that to delete entries in ports_vulnerable as well.

And how?

Via this table:

freshports.org=# \d commit_log_ports_vuxml
                             Table "public.commit_log_ports_vuxml"
    Column     |  Type   |                              Modifiers
---------------+---------+---------------------------------------------------------------------
 id            | integer | not null default nextval('commit_log_ports_vuxml_id_seq'::regclass)
 commit_log_id | integer | not null
 port_id       | integer | not null
 vuxml_id      | integer | not null
Indexes:
    "commit_log_ports_vuxml_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
    "$1" FOREIGN KEY (vuxml_id) REFERENCES vuxml(id) ON UPDATE CASCADE ON DELETE CASCADE
    "$2" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE CASCADE ON DELETE CASCADE
    "$3" FOREIGN KEY (commit_log_id) REFERENCES commit_log(id) ON UPDATE CASCADE ON DELETE CASCADE

freshports.org=#

When the vuxml entry is deleted, we can see all the ports affected by this vuln. We can then delete entries from the ports_vulnerable table as well.

Or so I think….

A number of new features have recently gone into FreshPorts. Each of which has been written above before. I will outline the changes, and provide a link to the original post about that feature.

• svn coming soon… - FreshPorts now handle commit from three mailing lists: cvs-doc, cvs-ports, and svn-src-all.
• What ports are dependant upon this port? - this feature has been available on beta and dev for some time. However, reading the comments for that post, it appears we are missing some items from the possible depends that we can pull in.

NOTE: the system is right now updating all the port dependencies. This might take a while… Your port may be out of date right now..

This issue was brought to my attention by wxs (Wesley Shields). The issue is evident at http://www.freshports.org/graphs2.php. When you get there, click on click on Commits Over Time by Committer and then wait for the page to load.

Scroll down to the D’s. There you’ll see a few committers suffixed with ‘(ports committer)’.

Looking in the database, we see:

$ psql freshports.org
psql (8.4.7)
Type "help" for help.

freshports.org=# select committer, message_id, commit_date from commit_log where committer like '%ports committer%' order by commit_date desc;
        committer         |                 message_id                  |      commit_date
--------------------------+---------------------------------------------+------------------------
 rea (ports committer)    | 201110280603.p9S63cVj045250@svn.freebsd.org | 2011-10-28 07:03:38+01
 eadler (ports committer) | 201110172131.p9HLV3xL041492@svn.freebsd.org | 2011-10-17 22:31:03+01
 eadler (ports committer) | 201110161430.p9GEUTFj066335@svn.freebsd.org | 2011-10-16 15:30:29+01
 eadler (ports committer) | 201110152106.p9FL684c030644@svn.freebsd.org | 2011-10-15 22:06:08+01
 rakuco (ports committer) | 201110132036.p9DKaiNi031421@svn.freebsd.org | 2011-10-13 21:36:44+01
 crees (ports committer)  | 201110081825.p98IP22D073560@svn.freebsd.org | 2011-10-08 19:25:02+01
 wxs (ports committer)    | 201110061613.p96GDlZk068019@svn.freebsd.org | 2011-10-06 17:13:47+01
 eadler (ports committer) | 201109282142.p8SLgDCV090941@svn.freebsd.org | 2011-09-28 22:42:13+01
 eadler (ports committer) | 201109282046.p8SKkqQT089293@svn.freebsd.org | 2011-09-28 21:46:52+01
 eadler (ports committer) | 201109281856.p8SIu2n2085649@svn.freebsd.org | 2011-09-28 19:56:02+01
 eadler (ports committer) | 201109281849.p8SInbrJ085366@svn.freebsd.org | 2011-09-28 19:49:37+01
 crees (ports committer)  | 201109281703.p8SH3n8r082164@svn.freebsd.org | 2011-09-28 18:03:49+01
 cs (ports committer)     | 201109162257.p8GMvowd042057@svn.freebsd.org | 2011-09-16 23:57:50+01
(13 rows)

freshports.org=#

Looking at one of these commits (the one by wxs), we see it is a source commit, as opposed to a port commit.

The original commit email contains that ‘(ports committer)’ text.

Ideally, the FreshPorts software would strip this value from the committer field before storing it in the database. It could also set a flag on that commit to highlight this particular situation.

It seems the fix is easy enough:

freshports.org=# update commit_log set committer = replace(committer, ' (ports committer)', '')  where committer like '%ports committer%';
UPDATE 13

But it seems the problem is more widespread than that:

freshports.org=# select committer, message_id, commit_date from commit_log where committer like '% committer%' order by commit_date desc;
       committer        |                 message_id                  |      commit_date
------------------------+---------------------------------------------+------------------------
 wblock (doc committer) | 201110140034.p9E0Yks5038994@svn.freebsd.org | 2011-10-14 01:34:46+01
 gjb (doc committer)    | 201110101123.p9ABNnDV057799@svn.freebsd.org | 2011-10-10 12:23:49+01
 gjb (doc committer)    | 201110101123.p9ABNJuE057749@svn.freebsd.org | 2011-10-10 12:23:19+01
 gjb (doc committer)    | 201110101115.p9ABFvON057412@svn.freebsd.org | 2011-10-10 12:15:57+01
 gjb (doc committer)    | 201110101114.p9ABEiVK057324@svn.freebsd.org | 2011-10-10 12:14:44+01
 gjb (doc committer)    | 201110101114.p9ABE3go057270@svn.freebsd.org | 2011-10-10 12:14:03+01
 gjb (doc committer)    | 201110101112.p9ABCejq057184@svn.freebsd.org | 2011-10-10 12:12:40+01
 gjb (doc committer)    | 201110101111.p9ABBx3e057123@svn.freebsd.org | 2011-10-10 12:11:59+01
 gjb (doc committer)    | 201110101111.p9ABB5M8057060@svn.freebsd.org | 2011-10-10 12:11:05+01
 gjb (doc committer)    | 201110101107.p9AB7anG056904@svn.freebsd.org | 2011-10-10 12:07:36+01
 gjb (doc committer)    | 201110101106.p9AB6nRm056835@svn.freebsd.org | 2011-10-10 12:06:49+01
 gjb (doc committer)    | 201110101105.p9AB5wIj056772@svn.freebsd.org | 2011-10-10 12:05:58+01
 gjb (doc committer)    | 201110092021.p99KL7kn025209@svn.freebsd.org | 2011-10-09 21:21:07+01
 gjb (doc committer)    | 201110021605.p92G5JXb070257@svn.freebsd.org | 2011-10-02 17:05:19+01
 gjb (doc committer)    | 201110012347.p91NlbkK038750@svn.freebsd.org | 2011-10-02 00:47:37+01
 gjb (doc committer)    | 201110012056.p91KuwwX033481@svn.freebsd.org | 2011-10-01 21:56:58+01
 wblock (doc committer) | 201109290337.p8T3bgwv002749@svn.freebsd.org | 2011-09-29 04:37:42+01
 gjb (doc committer)    | 201109290257.p8T2v86W001175@svn.freebsd.org | 2011-09-29 03:57:08+01
 gjb (doc committer)    | 201109290252.p8T2qXbO000985@svn.freebsd.org | 2011-09-29 03:52:33+01
 gjb (doc committer)    | 201109290229.p8T2TWYt000206@svn.freebsd.org | 2011-09-29 03:29:32+01
 gjb (doc committer)    | 201108310118.p7V1IN05074751@svn.freebsd.org | 2011-08-31 02:18:23+01
 gjb (doc committer)    | 201108310117.p7V1Hnam074693@svn.freebsd.org | 2011-08-31 02:17:49+01
 marck (doc committer)  | 201108301149.p7UBnMHr050082@svn.freebsd.org | 2011-08-30 12:49:22+01
 marck (doc committer)  | 201108301147.p7UBla7E049983@svn.freebsd.org | 2011-08-30 12:47:36+01
 gjb (doc committer)    | 201108241218.p7OCITW1053854@svn.freebsd.org | 2011-08-24 13:18:29+01
 marck (doc committer)  | 201108232025.p7NKPBBR018175@svn.freebsd.org | 2011-08-23 21:25:11+01
(26 rows)

freshports.org=#

In my previous post, I mentioned problems encountered with processing vuxml entries for horde4-imp. The problem was not specific to a particular port, but came to light because a vuln was registered for a port, and then later removed. I tracked down the cause to not clearing out statistics before processing the new vulnerabilities. This post shows the fix I have come up with.

The table which was not being updated properly was the ports_vulnerable table. This table looks like this:

freshports.org=# \d ports_vulnerable
    Table "public.ports_vulnerable"
 Column  |  Type   |     Modifiers
---------+---------+--------------------
 port_id | integer | not null
 current | integer | not null default 1
 past    | integer | not null default 0
Indexes:
    "ports_vulnerable_pkey" PRIMARY KEY, btree (port_id)
Foreign-key constraints:
    "$1" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE RESTRICT ON DELETE CASCADE

freshports.org=#

The issue was easily resolved with this patch:

--- scripts/process_vuxml.pl	2006/12/17 12:04:02	1.2
+++ scripts/process_vuxml.pl	2011/10/02 17:29:20	1.3
@@ -1,6 +1,6 @@
 #!/usr/bin/perl -w
 #
-# $Id: process_vuxml.pl,v 1.2 2006/12/17 12:04:02 dan Exp $
+# $Id: process_vuxml.pl,v 1.3 2011/10/02 17:29:20 dan Exp $
 #
 # Copyright (c) 2001-2004 DVL Software
 #
@@ -36,7 +36,13 @@ sub EmptyVuXML($) {
 	$sql = "DELETE FROM vuxml";
 	$sth = $dbh->prepare($sql);
 	if (!$sth->execute())  {
-		FreshPorts::Utilities::ReportError('warning', "Could not execute sql", 1);
+		FreshPorts::Utilities::ReportError('warning', "Could not execute sql: " . $sql, 1);
+	}
+
+	$sql = "DELETE FROM ports_vulnerable";
+	$sth = $dbh->prepare($sql);
+	if (!$sth->execute())  {
+		FreshPorts::Utilities::ReportError('warning', "Could not execute sql: " . $sql, 1);
 	}
 }
 

With patch, I’m clearing out the table each time a new vuln.xml file is processed. However, that is not enough. Whenever a record is removed from this table, the cache for that port needs to be updated.

The following post will queue up a cache deletion request for each delete on the ports_vulnerable table:

--- database-schema/ri.txt	2008/09/09 13:02:38	1.45
+++ database-schema/ri.txt	2011/10/02 17:30:40	1.46
@@ -1,5 +1,5 @@
 --
--- $Id: ri.txt,v 1.45 2008/09/09 13:02:38 dan Exp $
+-- $Id: ri.txt,v 1.46 2011/10/02 17:30:40 dan Exp $
 --
 -- Copyright (c) 1998-2007 DVL Software Limited
 --
@@ -254,6 +254,42 @@ CREATE TRIGGER ports_clear_cache
     EXECUTE PROCEDURE ports_clear_cache();

+CREATE OR REPLACE FUNCTION ports_vulnerable_delete_clear_cache() RETURNS TRIGGER AS $$
+   DECLARE
+      l_cache_clearing_ports_id   int8;
+      l_port      text;
+      l_category  text;
+   BEGIN
+		IF TG_OP = 'DELETE' THEN
+         SELECT port_id
+           INTO l_cache_clearing_ports_id
+           FROM cache_clearing_ports
+          WHERE port_id = OLD.port_id;
+
+          IF NOT FOUND THEN
+            SELECT category, name
+              INTO l_category, l_port
+              FROM ports_all
+             WHERE id = OLD.port_id;
+            INSERT INTO cache_clearing_ports (port_id, category, port)
+            VALUES (OLD.port_id, l_category, l_port);
+          END IF;
+
+          NOTIFY port_updated;
+    END IF;
+
+      -- when a port changes, add an entry to the cache clearing table
+      RETURN OLD;
+   END
+$$ LANGUAGE 'plpgsql';
+
+  DROP TRIGGER ports_vulnerable_delete_clear_cache ON ports_vulnerable;
+CREATE TRIGGER ports_vulnerable_delete_clear_cache
+    AFTER DELETE on ports_vulnerable
+    FOR EACH ROW
+    EXECUTE PROCEDURE ports_vulnerable_delete_clear_cache();
+
+

I’ve tried this in dev, and in production, and staging. It seems to be the right thing to do.

Mr Thomas wrote me regarding mail/horde4-imp and a particular vulnerability. Specifically, he figured that a recent commit (link to mail archives) fixed a bad entry in vuxml. Let’s look into that now.

I first compared dev, beta, and production. All three had the port flagged as vulnerable. This situation is indicated by a skull at the top of the page, in the ‘Port details’ section. However, only production listed any particular revisions/commits as being vulnerable. This is indicated by skull beside the commit date / revision number. Production also listed versions > 5.0.3 as vulnerable. None of this matches the definition of the vulnerability at the vuxml site, which clearly states:

• 4.2,1 < horde-imp < 4.3.8,1
• horde-imp < 4.3.8

Let’s look at vuln.xml:

<name>horde-imp</name>
<range><gt>4.2,1</gt><lt>4.3.8,1</lt></range>
<range><lt>4.3.8</lt></range>

The next check: has FreshPorts processed the latest vuxml file?

Clicking on the skull takes us to http://www.freshports.org/vuxml.php?package=horde-imp, where you can see:

The last vuln.xml file processed by FreshPorts is:

    Revision:  1.2451
    Date:      2011/09/23
    Time:      20:02:19
    Committer: delphij

If we compare that to the most recent commit for security/vuxml, we can confirm the revision number. Yes, that matches up.

That looks right.

OK, what ports have a package name of horde-imp?

freshports.org=# select id, category || '/' || name as port from ports_active where package_name = 'horde-imp';
  id   |      port
-------+-----------------
 21107 | mail/horde-imp
 28743 | mail/horde4-imp
(2 rows)

freshports.org=#

This confirms that this vuxml entry affects both horde4-imp and horde-imp. However, we are only interested in id = 28743.

Looking in the log for the vuxml processing, I found nothing to indicate that 28743 is vulnerable to anything. Up to this point, I was doing everything on the dev server. Next, I went to the production server and checked the vuxml log. Nothing there either. OK, so perhaps this is something else entirely.

Next, I went to production and cleared the cache for all mail/horde* ports. Ahah! Now prod, dev, and test are in agreement.

The only issue now is the vulnerability skull at the top of the page. This usually indicates that a vulnerability affects the latest revision of the port. Hmmm, this needs further investigation.

This query shows that FreshPorts thinks there is a current vulnerability:

freshports.org=# select * from ports_vulnerable where port_id = 28743;
 port_id | current | past
---------+---------+------
   28743 |       1 |    0
(1 row)

freshports.org=#

As a test, I reran, with a commit, the adjusting function:

freshports.org=# select * from ports_vulnerable where port_id = 28743;
 port_id | current | past
---------+---------+------
   28743 |       1 |    0
(1 row)

freshports.org=# begin;
BEGIN
freshports.org=# select PortsVulnerabilityCountAdjust(28743);
 portsvulnerabilitycountadjust
-------------------------------
                             0
(1 row)

freshports.org=# select * from ports_vulnerable where port_id = 28743;
 port_id | current | past
---------+---------+------
(0 rows)

freshports.org=# rollback;
ROLLBACK
freshports.org=# select * from ports_vulnerable where port_id = 28743;
 port_id | current | past
---------+---------+------
   28743 |       1 |    0
(1 row)

freshports.org=#

My conclusion: PortsVulnerabilityCountAdjust() was not invoked for this port.

The vuxml log confirms this.

Offhand, I think the solution is to delete everything from ports_vulnerable each time we process a new vuxml file.

That’s all for tonight. I want to think about this for a while longer.

In short, the concept of any computer system is usually straight forward. The complexity comes from the exceptions. Take commits for example. Yes, each commit affects a file. Or so you’d think. Consider this commit.

FreshPorts parses the email. In that email, there are no files mentioned. This caused a problem during the loading. FreshPorts, as a sanity test, complained there were no files. This complaint is now just a notification, not a fatal error.

The fix for that was rather simple:

-		FreshPorts::Utilities::ReportError('Err', "No files found in commit '$Updates{MessageId}'.  Has someone done a cvs import instead of addport?", 1)
+		FreshPorts::Utilities::ReportError('Err', "No files found in commit '$Updates{MessageId}'.  Has someone done a cvs import instead of addport?", 0)

In this case, the 1 means die, fatal error. Hmmm, wouldn’t that be better with a constant?

The next issue was the display of the commit. Production was displaying the commit. Dev and beta was not. Eventually I figured out the problem was a long-standing stored procedure.

The fix for that issue was much more subtle:

@@ -2185,13 +2185,14 @@ SELECT 0 AS port_id,
        NULL::text AS only_for_archs,
        NULL::text AS not_for_archs,
        OnWatchList($4, CLE.element_id) AS watch
-  FROM commit_log CL, commit_log_elements CLE, element E
- WHERE CL.id          = CLE.commit_log_id
-   AND CL.message_id  = $1
-   AND CLE.element_id = E.id
+  FROM commit_log CL LEFT OUTER JOIN commit_log_elements CLE ON (CL.id = CLE.commit_log_id)
+       LEFT OUTER JOIN element E ON (CLE.element_id = E.id)
+ WHERE CL.message_id  = $1
 ORDER BY port, element_pathname
    LIMIT $2
   OFFSET $3;
+
+  -- NOTE some commits touch nothing....  e.g. 201109230051.p8N0pbV2045995@svn.freebsd.org

 $$ LANGUAGE SQL STABLE;

This is the code for retrieving a commit. It also retrieves the files touched by that commit. Thus, if there are no files, it will not retrieve anything. Thereby, incorrectly reporting ‘Sorry, nothing found in the database….’.

The old code assumed that for every entry (commit) in the commit_log table, there would be at least one element (file). The SQL change made entries in the commit_log_elements and the element tables would be optional.

This commit now displays correctly in all three environments:

1. Production
2. Beta
3. dev

But why was production displaying the commit? Because it still contains older code which has since been fixed. This fix stopped FreshPorts from incorrectly treating ‘Directory Properties:’ and ‘(props changed)’ as ‘files”. The fix was on beta and dev. Hence, those systems had no entries in commit_log_elements for those bogus files. This is why they would not display the commit.

But now I have bogus files sitting around. I deleted them by hand.

Then I noticed that commit_log_elements was missing foreign keys. I added them, but first had to delete rows which referred to invalid references. These two commands took care of that:

delete from commit_log_ports_elements where not exists (select * from commit_log where commit_log_id = id);
delete from commit_log_ports_elements where not exists (select * from element where element_id = id);

Then I added two new foreign keys:

alter table commit_log_ports_elements
    add foreign key (element_id)
       references element(id) on update cascade on delete cascade;

alter table commit_log_ports_elements
    add foreign key (commit_log_id)
       references commit_log(id) on update cascade on delete cascade;

The table in question now looks like this:

freshports.org=# \d commit_log_elements
                               Table "public.commit_log_elements"
    Column     |     Type     |                            Modifiers
---------------+--------------+------------------------------------------------------------------
 id            | integer      | not null default nextval('commit_log_elements_id_seq'::regclass)
 commit_log_id | integer      | not null
 element_id    | integer      | not null
 revision_name | text         |
 change_type   | character(1) | not null
Indexes:
    "commit_log_elements_pkey" PRIMARY KEY, btree (id)
    "commit_log_elements_clid" btree (commit_log_id)
    "commit_log_elements_ei" btree (element_id)
Check constraints:
    "commit_log_elements_change_type" CHECK (change_type = 'A'::bpchar OR change_type = 'M'::bpchar OR change_type = 'R'::bpchar)
Foreign-key constraints:
    "$1" FOREIGN KEY (commit_log_id) REFERENCES commit_log(id) ON UPDATE CASCADE ON DELETE CASCADE
    "$2" FOREIGN KEY (element_id, revision_name) REFERENCES element_revision(element_id, revision_name) ON UPDATE CASCADE ON DELETE CASCADE
Referenced by:
    TABLE "commit_log_port_elements" CONSTRAINT "$2" FOREIGN KEY (commit_log_element_id) REFERENCES commit_log_elements(id) ON UPDATE CASCADE ON DELETE CASCADE
Triggers:
    element_delete_check BEFORE INSERT OR UPDATE ON commit_log_elements FOR EACH ROW EXECUTE PROCEDURE element_delete_check()

Now I should be able to get into other things. :)

A while back, the src tree for FreeBSD moved to svn. The ports tree is still on cvs. For some time now, http://svn.freshports.org/ has been including the svn commits. But now, as old mailing lists become inactive, it’s time to start integrating things into the main websites.

The dev and beta websites have been running on three different mailing lists and two repository types for a few months now. This sounds complicated, but because all of FreshPorts works off an XML-based input, all that’s needed for input is a script to convert the email commit to the XML format. From there, all commits are the same to FreshPorts.

FreshPorts now uses three mailing lists. If you think we should cover more, we can.

1. cvs-doc : CVS commit messages for the doc and www trees
2. cvs-ports : CVS commit messages for the ports tree
3. svn-src-all : SVN commit messages for the entire src tree (except for “user” and “projects”)

Tonight I’ll be setting up a new FreshPorts server and going through the upgrade process to make sure all goes well.

As mentioned in the comments of my previous post, some commits have been processed with an incorrect prefix. That is, some pathnames are missing the src/ part of the path.

I just went looking…

I found interesting stuff. 1233 rows of interesting stuff.

For example:

SELECT *
  FROM element_pathname
 WHERE pathname NOT LIKE '/ports/%'
   AND pathname NOT LIKE '/www/%'
   AND pathname NOT LIKE '/src/%'
   AND pathname NOT LIKE '/doc/%'
   AND pathname NOT LIKE '/projects/%'
   AND pathname NOT LIKE '/CVSROOT%'
ORDER BY PATHNAME;

…gives me:

 element_id |                                            pathname
------------+-------------------------------------------------------------------------------------------------
     374165 | /Directory Properties:
     374343 | /bind9
     374605 | /bind9/9.4-ESV-R5
     374508 | /bind9/9.6-ESV-R5
     374517 | /bind9/dist-9.4

Other interesting tidbits include (these listings are not complete):

    374260 | /cddl/contrib
     374261 | /cddl/contrib/opensolaris
     374287 | /cddl/contrib/opensolaris/   (props changed)
     374262 | /cddl/contrib/opensolaris/cmd
     374263 | /cddl/contrib/opensolaris/cmd/zfs
     374264 | /cddl/contrib/opensolaris/cmd/zfs/zfs.8

and:

     373976 | /contrib
     374613 | /contrib/bind9
     374776 | /contrib/bind9/   (props changed)
     374618 | /contrib/bind9/CHANGES
     374619 | /contrib/bind9/COPYRIGHT
     374620 | /contrib/bind9/Makefile.in
     374621 | /contrib/bind9/README

And yet more:

     374781 | /contrib/gcc/c-decl.c
     374782 | /contrib/gcc/c.opt
     374783 | /contrib/gcc/common.opt
...
     375026 | /contrib/gdb
     375027 | /contrib/gdb/FREEBSD-diffs
     375028 | /contrib/gdb/gdb
     375029 | /contrib/gdb/gdb/sparc64fbsd-tdep.c
...
     374871 | /crypto/openssh/kex.c
     374902 | /crypto/openssh/kex.c   (props changed)
     374872 | /crypto/openssh/kex.h
     374903 | /crypto/openssh/kex.h   (props changed)
     374873 | /crypto/openssh/misc.c
     374904 | /crypto/openssh/misc.c   (props changed)
     374874 | /crypto/openssh/misc.h
...
      78952 | /distrib/cvsup
      78953 | /distrib/cvsup/sup
     120661 | /distrib/cvsup/sup/README
      95220 | /distrib/cvsup/sup/cvs-all
      95221 | /distrib/cvsup/sup/cvs-all/list.cvs
     131987 | /distrib/cvsup/sup/cvsroot-all
...
     375030 | /gnu
     375031 | /gnu/usr.bin
     375032 | /gnu/usr.bin/gdb
     375033 | /gnu/usr.bin/gdb/arch
     375034 | /gnu/usr.bin/gdb/arch/sparc64
     375035 | /gnu/usr.bin/gdb/arch/sparc64/Makefile
     375036 | /gnu/usr.bin/gdb/arch/sparc64/init.c
...
     374509 | /lib/bind
     374606 | /lib/bind/config.h
     374510 | /lib/bind/dns
     374607 | /lib/bind/dns/code.h
     374511 | /lib/bind/dns/dns
...
     373994 | /lib/libarchive
     374166 | /lib/libarchive/   (props changed)
     374048 | /lib/libarchive/Makefile
     374049 | /lib/libarchive/README
...
     374913 | /lib/libc
     374914 | /lib/libc/net
     375040 | /lib/libc/net/ethers.3
     374915 | /lib/libc/net/sctp_sys_calls.c
     375013 | /lib/libkvm
     375014 | /lib/libkvm/kvm_ia64.c
     374313 | /lib/libmemstat
     374314 | /lib/libmemstat/libmemstat.3
     374315 | /lib/libmemstat/memstat.c
...
     375021 | /lib/libthread_db/arch
     375022 | /lib/libthread_db/arch/amd64
     375023 | /lib/libthread_db/arch/amd64/libpthread_md.c
...
     374964 | /release/picobsd/   (props changed)
     374965 | /release/picobsd/floppy.tree
     374966 | /release/picobsd/floppy.tree/sbin
     374967 | /release/picobsd/floppy.tree/sbin/   (props changed)
     374968 | /release/picobsd/floppy.tree/sbin/dhclient-script   (props changed)
...
     375010 | /sbin/rcorder/rcorder.8
     374309 | /share
     374777 | /share/doc
     374778 | /share/doc/bind9
...
     375072 | /sys/arm/conf
     375073 | /sys/arm/conf/AVILA
     375074 | /sys/arm/conf/BWCT
     375075 | /sys/arm/conf/CAMBRIA
     375076 | /sys/arm/conf/CNS11XXNAS
...
     374985 | /tools/regression/security/cap_test/cap_test_sysctl.c
     373980 | /usr.bin
     374975 | /usr.bin/ar
     374976 | /usr.bin/ar/write.c

I’m pretty sure there’s a simple bug fix for this. But my main priority now is finding a job.

Alex wrote in to mention that FreshSource was missing some src commits. In fact, he wrote twice. I’m glad he did. I replied to his first post saying that I was seeing www commits… He pointed out that no src commits were turning up.

OH!

I checked my local mailing list archive (i.e. a folder in Mail.app). The last src commit I saw was 31 July. OK, that sounds like a switch was thrown on 1 Aug… Yes. FreshPorts had still been using the old list.

List-Id: **OBSOLETE** CVS commit messages for the src tree
        <cvs-src-old.freebsd.org>

However, code exists for parsing the new mailing list:

List-Id: SVN commit messages for the src tree for head/-current
        <svn-src-head.freebsd.org>

Compare these two URLs and you’ll see that the latter contains the missing src commits:

• http://www.freshsource.org/commits.php
• http://svn.freshsource.org/commits.php

These two websites use different ~/.procmail rules and scripts to get the data into the database. They also use two different databases. I’ll match up the rules and scripts, then throw the missing svn commit emails at them.

Lately, I’ve been thinking and writing about how to list ports that depend upon this port. Today, another special situation arose: doing commits out of order.

Case in question: this commit support for databases/php5-sqlite3 to lang/php5-extensions.

The lang/php5-extensions commit was at 15:52:14. However, databases/php5-sqlite3 was not created until 15:54:29.

I am not sure how to best handle such situations.