Dec 272013
 

It’s pretty easy to say that there’s just one branch for the FreeBSD ports tree. And it’s not really a branch at all. Commits were done to HEAD, and branches were ignored. That has recently changed. The FreeBSD ports tree will be branched every three months and commits to that branch will be tightly controlled. HEAD will continue along as it has in the past.

System background

Before I get to how we need to change FreshPorts, let’s see what it is doing now.

The FreshPorts website was designed with HEAD in mind. However, the FreshPorts database was designed with branches in mind. FreshPorts currently tracks 124 different branches:

freshports.org=# \d system_branch
                            Table "public.system_branch"
   Column    |  Type   |                         Modifiers                          
-------------+---------+------------------------------------------------------------
 id          | integer | not null default nextval('system_branch_id_seq'::regclass)
 system_id   | integer | not null
 branch_name | text    | 
Indexes:
    "system_branch_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
    "$1" FOREIGN KEY (system_id) REFERENCES system(id) ON UPDATE CASCADE ON DELETE CASCADE
Referenced by:
    TABLE "system_branch_element_revision" CONSTRAINT "$2" FOREIGN KEY (system_branch_id) REFERENCES system_branch(id) ON UPDATE CASCADE ON DELETE CASCADE

freshports.org=# select * from system_branch order by 2;
 id  | system_id |           branch_name            
-----+-----------+----------------------------------
   1 |         1 | HEAD
   2 |         1 | RELENG_4
   3 |         1 | ZLIB
   4 |         1 | NETGRAPH
   5 |         1 | LUKEM
   6 |         1 | OPENPAM
   7 |         1 | INTEL
   8 |         1 | BELL_LABS
   9 |         1 | LWALL
  10 |         1 | NETBSD
  11 |         1 | RELENG_3
  12 |         1 | OpenBSD
  13 |         1 | DARRENR
  14 |         1 | FSF
  15 |         1 | NRL
  16 |         1 | RELENG_4_5
  17 |         1 | RELENG_4_4
  18 |         1 | RELENG_4_3
  19 |         1 | ISC
  20 |         1 | 1.1.1
  21 |         1 | SOLAR_DESIGNER
  22 |         1 | ADO
  23 |         1 | WIP_GCC31
  24 |         1 | SENDMAIL
  25 |         1 | KNU
  26 |         1 | HEAD,RELENG_4
  27 |         1 | OPENBSD
  28 |         1 | MAIN
  29 |         1 | BP
  30 |         1 | FREEBSD
  31 |         1 | NCURSES
  32 |         1 | RELENG_4_6
  33 |         1 | TCPDUMP_ORG
  34 |         1 | OPENSSH
  35 |         1 | NetBSD
  36 |         1 | CVSWEB
  37 |         1 | RELENG_2_2
  38 |         1 | 1
  39 |         1 | JPO
  40 |         1 | ZOULAS
  41 |         1 | LBL
  42 |         1 | OPENSSL
  43 |         1 | rel-2_0-branch
  44 |         1 | RELENG_4_4,RELENG_4_5,RELENG_4_6
  45 |         1 | KTH
  46 |         1 | CVSHOME
  47 |         1 | EXPAT
  48 |         1 | RELENG_4_7
  49 |         1 | TEXFM
  50 |         1 | maho
  51 |         1 | DMILLS
  52 |         1 | KORTY
  53 |         1 | RELENG_5_0
  54 |         1 | gmt
  55 |         1 | LINUX
  56 |         1 | davidxu
  57 |         1 | NETLIB
  58 |         1 | RELENG_4_8
  59 |         1 | CREATIVE
  60 |         1 | LIBUWX
  61 |         1 | COLUMBIA
  62 |         1 | RELENG_5_1
  63 |         1 | ALSA
  64 |         1 | Atheros
  65 |         1 | GNU
  66 |         1 | LIBTHR_ALPHA_INIT
  67 |         1 | ATHEROS
  68 |         1 | SNERT
  69 |         1 | BEGEMOT
  70 |         1 | RELENG_4,RELENG_4_8
  71 |         1 | RELENG_4_9
  72 |         1 | RELENG_3,RELENG_4
  73 |         1 | DARRENREED
  74 |         1 | RELENG_5_2
  75 |         1 | LINUX_TEST_PROJECT
  76 |         1 | OPEN_POSIX_TESTSUITE
  77 |         1 | MTM_LOCAL
  78 |         1 | CYCLIC
  79 |         1 | VJS
  80 |         1 | LESS
  81 |         1 | RELENG_4_10
  82 |         1 | JOERG
  83 |         1 | KAME
  84 |         1 | RELENG_5
  85 |         1 | HEAD,RELENG_5
  86 |         1 | RELENG_5_3
  87 |         1 | RELENG_4_11
  88 |         1 | NVIDIA
  89 |         1 | RELENG_5_4
  90 |         1 | netbsd
  91 |         1 | BZIP2
  92 |         1 | MALINEN
  93 |         1 | RELENG_6
  94 |         1 | RELENG_6_0
  95 |         1 | TrustedBSD
  96 |         1 | CSUP
  97 |         1 | RELENG_6_1
  98 |         1 | RELENG_5_5
  99 |         1 | bsd
 100 |         1 | ATHEROS,HEAD
 101 |         1 | FSF,HEAD
 102 |         1 | mackerras
 103 |         1 | RELENG_6_2
 104 |         1 | GSF
 105 |         1 | FreeBSD
 106 |         1 | RELENG_7
 107 |         1 | RELENG_6_3
 108 |         1 | RELENG_7_0
 109 |         1 | HEAD,INTEL
 110 |         1 | 2.0
 111 |         1 | SUN
 112 |         1 | RELENG_6_4
 113 |         1 | RELENG_7_1
 114 |         1 | RELENG_7_2
 115 |         1 | RELENG_8
 116 |         1 | RELENG_8_0
 117 |         1 | RELENG_7_3
 118 |         1 | RELENG_8_1
 119 |         1 | RELENG_8_2
 120 |         1 | RELENG_7_4
 121 |         1 | UNKNOWN
 122 |         1 | VENDOR
 123 |         1 | RELENG_9
 124 |         1 | RELENG_10
(124 rows)

freshports.org=# 

You will notice that some commits are to multiple branches. e.g. HEAD,RELENG_4. That should be cleaned up so the system puts that incoming commit into two different branches, not one.

Why are all the system_id values set to 1? That is easily answered:

freshports.org=#  \d system
                            Table "public.system"
   Column    |   Type   |                      Modifiers                      
-------------+----------+-----------------------------------------------------
 id          | integer  | not null default nextval('system_id_seq'::regclass)
 name        | text     | not null
 time_adjust | interval | not null default '00:00:00'::interval
Indexes:
    "system_pkey" PRIMARY KEY, btree (id)
Referenced by:
    TABLE "commit_log" CONSTRAINT "$1" FOREIGN KEY (system_id) REFERENCES system(id) ON UPDATE CASCADE ON DELETE CASCADE
    TABLE "system_branch" CONSTRAINT "$1" FOREIGN KEY (system_id) REFERENCES system(id) ON UPDATE CASCADE ON DELETE CASCADE

freshports.org=# select * from system;
 id |  name   | time_adjust 
----+---------+-------------
  1 | FreeBSD | 00:00:00
(1 row)

freshports.org=# 

The database is designed to hold commits from various systems. Only FreeBSD has been added so far.

So, FreshPorts is tracking branches for all commits to the FreeBSD repo, not just ports.

Testing the branch theory

To see if I could backtrack from a branch to the commits on the branch, I started with a recent commit to sysutils/zfs-periodic (a port I hope to make use of very soon now). Let’s start with this query:

freshports.org=# SELECT CL.id
  FROM commit_log CL
 WHERE CL.message_id = E'201312271836.rBRIaoHN063395@svn.freebsd.org';
   id   
--------
 500046
(1 row)

freshports.org=# 

That’s the unique id for this commit. What files were touched by that commit?

freshports.org=# SELECT CL.id, CLE.*, element_pathname(element_id)
  FROM commit_log CL, commit_log_elements CLE
 WHERE CL.message_id = E'201312271836.rBRIaoHN063395@svn.freebsd.org'
   AND CLE.commit_log_id = CL.id;
   id   |   id    | commit_log_id | element_id | revision_name | change_type |              element_pathname              
--------+---------+---------------+------------+---------------+-------------+--------------------------------------------
 500046 | 2269113 |        500046 |     359439 | 337778        | M           | /ports/head/sysutils/zfs-periodic/Makefile
(1 row)

freshports.org=# 

OK. Let’s connect into the system_branch_element_revision table and see what references we find:

freshports.org=# SELECT CL.id, CLE.*, element_pathname(CLE.element_id)
                   FROM commit_log CL, commit_log_elements CLE, system_branch_element_revision SBER
                  WHERE CL.message_id = E'201312271836.rBRIaoHN063395@svn.freebsd.org'
                    AND CLE.commit_log_id = CL.id
                    AND CLE.element_id    = SBER.element_id
                    AND CLE.revision_name = SBER.revision_name;
 id | id | commit_log_id | element_id | revision_name | change_type | element_pathname 
----+----+---------------+------------+---------------+-------------+------------------
(0 rows)

freshports.org=#

Nothing. Hmmm. That’s bad.

At this point, I start playing with various queries to see how many references I could find to the element_id in question.

freshports.org=# select * 
                   from system_branch_element_revision SBER
                  WHERE 359439 = SBER.element_id;
 system_branch_id | element_id | revision_name 
------------------+------------+---------------
                1 |     359439 | 1.1
              121 |     359439 | 301858
(2 rows)

freshports.org=# select * from commit_log_elements CLE
                  WHERE 359439 = CLE.element_id;
   id    | commit_log_id | element_id | revision_name | change_type 
---------+---------------+------------+---------------+-------------
 2269113 |        500046 |     359439 | 337778        | M
 2190381 |        486227 |     359439 | 327772        | M
 1611161 |        387853 |     359439 | 1.1           | A
 1790229 |        421143 |     359439 | 1.2           | M
 1917252 |        443752 |     359439 | 301858        | M
 1987472 |        455607 |     359439 | 308502        | M
 2018932 |        462046 |     359439 | 312371        | M
(7 rows)

freshports.org=# 

There are 7 commits to the Makefile, but only two are recorded in the system_branch_element_revision. That’s not good. We want them all recorded.

Where is it going wrong?

I searched the code for references to system_branch_element_revision and found this stored procedure:

CREATE OR REPLACE FUNCTION ElementTagSet(int4, int4, text) RETURNS boolean AS $$
   DECLARE
      SystemBranchID  ALIAS for $1;
      ElementID       ALIAS for $2;
      RevisionName    ALIAS for $3;

      ExistingRevisionFound text;

   BEGIN  

-- has this element already been tagged with this version?
      ExistingRevisionFound := RevisionForTagGet(SystemBranchID, ElementID);

      IF ExistingRevisionFound IS NULL THEN
         insert into system_branch_element_revision
                (system_branch_id, element_id, revision_name)
          values (SystemBranchID, ElementID, RevisionName);
      END IF;

      return 1;

   END;
$$
    LANGUAGE plpgsql; 

I also found log output which showed that the example commit executed this SQL (note, the previous queries were on the production database; the following queries are on the test database; thus, the element_id in question may vary):

sql = 'select ElementTagSet(1, 359439, '337778')'

Let’s see what the above stored procedure does with that input.

freshports.org=# begin;
BEGIN
freshports.org=# select ElementTagSet(1, 359439, '337778');
 elementtagset 
---------------
 t
(1 row)

freshports.org=# select * from system_branch_element_revision where revision_name = '337778';
 system_branch_id | element_id | revision_name 
------------------+------------+---------------
(0 rows)

freshports.org=# rollback;
ROLLBACK
freshports.org=# 

Clearly, that’s not working as designed.

Digging deeper

Let’s see what RevisionForTagGet() returns. This function is straight forward:

CREATE OR REPLACE FUNCTION RevisionForTagGet(int4, int4 ) RETURNS text AS $$
   DECLARE
      SystemBranchID   ALIAS for $1;
      ElementID        ALIAS for $2;
      TempRevisionName text;

-- for a given Element and TagName (System version id)
-- return the revision name assocated with it.
                            
   BEGIN
      select revision_name
        into TempRevisionName
        from system_branch_element_revision
       where system_branch_id = SystemBranchID
         and element_id       = ElementID;

      return TempRevisionName;
   END;
$$ LANGUAGE 'plpgsql';



freshports.org=# select RevisionForTagGet(1, 359439);
 revisionfortagget 
-------------------
 1.1
(1 row)

freshports.org=# 

Looking at ElementTagSet() on line 14, I think that’s the key. What we need to do is an insert, and that insert is not happening. I think the cause is the ElementTagSet() function, which needs to do an insert if the tuple is not already in the table. That probably means that RevisionForTagGet() is no longer required. I think the fix is pretty straight forward, but I’ve run out of time today.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive