It’s pretty easy to say that there’s just one branch for the FreeBSD ports tree. And it’s not really a branch at all. Commits were done to HEAD, and branches were ignored. That has recently changed. The FreeBSD ports tree will be branched every three months and commits to that branch will be tightly controlled. HEAD will continue along as it has in the past.
System background
Before I get to how we need to change FreshPorts, let’s see what it is doing now.
The FreshPorts website was designed with HEAD in mind. However, the FreshPorts database was designed with branches in mind. FreshPorts currently tracks 124 different branches:
freshports.org=# \d system_branch
Table "public.system_branch"
Column | Type | Modifiers
-------------+---------+------------------------------------------------------------
id | integer | not null default nextval('system_branch_id_seq'::regclass)
system_id | integer | not null
branch_name | text |
Indexes:
"system_branch_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"$1" FOREIGN KEY (system_id) REFERENCES system(id) ON UPDATE CASCADE ON DELETE CASCADE
Referenced by:
TABLE "system_branch_element_revision" CONSTRAINT "$2" FOREIGN KEY (system_branch_id) REFERENCES system_branch(id) ON UPDATE CASCADE ON DELETE CASCADE
freshports.org=# select * from system_branch order by 2;
id | system_id | branch_name
-----+-----------+----------------------------------
1 | 1 | HEAD
2 | 1 | RELENG_4
3 | 1 | ZLIB
4 | 1 | NETGRAPH
5 | 1 | LUKEM
6 | 1 | OPENPAM
7 | 1 | INTEL
8 | 1 | BELL_LABS
9 | 1 | LWALL
10 | 1 | NETBSD
11 | 1 | RELENG_3
12 | 1 | OpenBSD
13 | 1 | DARRENR
14 | 1 | FSF
15 | 1 | NRL
16 | 1 | RELENG_4_5
17 | 1 | RELENG_4_4
18 | 1 | RELENG_4_3
19 | 1 | ISC
20 | 1 | 1.1.1
21 | 1 | SOLAR_DESIGNER
22 | 1 | ADO
23 | 1 | WIP_GCC31
24 | 1 | SENDMAIL
25 | 1 | KNU
26 | 1 | HEAD,RELENG_4
27 | 1 | OPENBSD
28 | 1 | MAIN
29 | 1 | BP
30 | 1 | FREEBSD
31 | 1 | NCURSES
32 | 1 | RELENG_4_6
33 | 1 | TCPDUMP_ORG
34 | 1 | OPENSSH
35 | 1 | NetBSD
36 | 1 | CVSWEB
37 | 1 | RELENG_2_2
38 | 1 | 1
39 | 1 | JPO
40 | 1 | ZOULAS
41 | 1 | LBL
42 | 1 | OPENSSL
43 | 1 | rel-2_0-branch
44 | 1 | RELENG_4_4,RELENG_4_5,RELENG_4_6
45 | 1 | KTH
46 | 1 | CVSHOME
47 | 1 | EXPAT
48 | 1 | RELENG_4_7
49 | 1 | TEXFM
50 | 1 | maho
51 | 1 | DMILLS
52 | 1 | KORTY
53 | 1 | RELENG_5_0
54 | 1 | gmt
55 | 1 | LINUX
56 | 1 | davidxu
57 | 1 | NETLIB
58 | 1 | RELENG_4_8
59 | 1 | CREATIVE
60 | 1 | LIBUWX
61 | 1 | COLUMBIA
62 | 1 | RELENG_5_1
63 | 1 | ALSA
64 | 1 | Atheros
65 | 1 | GNU
66 | 1 | LIBTHR_ALPHA_INIT
67 | 1 | ATHEROS
68 | 1 | SNERT
69 | 1 | BEGEMOT
70 | 1 | RELENG_4,RELENG_4_8
71 | 1 | RELENG_4_9
72 | 1 | RELENG_3,RELENG_4
73 | 1 | DARRENREED
74 | 1 | RELENG_5_2
75 | 1 | LINUX_TEST_PROJECT
76 | 1 | OPEN_POSIX_TESTSUITE
77 | 1 | MTM_LOCAL
78 | 1 | CYCLIC
79 | 1 | VJS
80 | 1 | LESS
81 | 1 | RELENG_4_10
82 | 1 | JOERG
83 | 1 | KAME
84 | 1 | RELENG_5
85 | 1 | HEAD,RELENG_5
86 | 1 | RELENG_5_3
87 | 1 | RELENG_4_11
88 | 1 | NVIDIA
89 | 1 | RELENG_5_4
90 | 1 | netbsd
91 | 1 | BZIP2
92 | 1 | MALINEN
93 | 1 | RELENG_6
94 | 1 | RELENG_6_0
95 | 1 | TrustedBSD
96 | 1 | CSUP
97 | 1 | RELENG_6_1
98 | 1 | RELENG_5_5
99 | 1 | bsd
100 | 1 | ATHEROS,HEAD
101 | 1 | FSF,HEAD
102 | 1 | mackerras
103 | 1 | RELENG_6_2
104 | 1 | GSF
105 | 1 | FreeBSD
106 | 1 | RELENG_7
107 | 1 | RELENG_6_3
108 | 1 | RELENG_7_0
109 | 1 | HEAD,INTEL
110 | 1 | 2.0
111 | 1 | SUN
112 | 1 | RELENG_6_4
113 | 1 | RELENG_7_1
114 | 1 | RELENG_7_2
115 | 1 | RELENG_8
116 | 1 | RELENG_8_0
117 | 1 | RELENG_7_3
118 | 1 | RELENG_8_1
119 | 1 | RELENG_8_2
120 | 1 | RELENG_7_4
121 | 1 | UNKNOWN
122 | 1 | VENDOR
123 | 1 | RELENG_9
124 | 1 | RELENG_10
(124 rows)
freshports.org=#
You will notice that some commits are to multiple branches. e.g. HEAD,RELENG_4. That should be cleaned up so the system puts that incoming commit into two different branches, not one.
Why are all the system_id values set to 1? That is easily answered:
freshports.org=# \d system
Table "public.system"
Column | Type | Modifiers
-------------+----------+-----------------------------------------------------
id | integer | not null default nextval('system_id_seq'::regclass)
name | text | not null
time_adjust | interval | not null default '00:00:00'::interval
Indexes:
"system_pkey" PRIMARY KEY, btree (id)
Referenced by:
TABLE "commit_log" CONSTRAINT "$1" FOREIGN KEY (system_id) REFERENCES system(id) ON UPDATE CASCADE ON DELETE CASCADE
TABLE "system_branch" CONSTRAINT "$1" FOREIGN KEY (system_id) REFERENCES system(id) ON UPDATE CASCADE ON DELETE CASCADE
freshports.org=# select * from system;
id | name | time_adjust
----+---------+-------------
1 | FreeBSD | 00:00:00
(1 row)
freshports.org=#
The database is designed to hold commits from various systems. Only FreeBSD has been added so far.
So, FreshPorts is tracking branches for all commits to the FreeBSD repo, not just ports.
Testing the branch theory
To see if I could backtrack from a branch to the commits on the branch, I started with a recent commit to sysutils/zfs-periodic (a port I hope to make use of very soon now). Let’s start with this query:
freshports.org=# SELECT CL.id FROM commit_log CL WHERE CL.message_id = E'201312271836.rBRIaoHN063395@svn.freebsd.org'; id -------- 500046 (1 row) freshports.org=#
That’s the unique id for this commit. What files were touched by that commit?
freshports.org=# SELECT CL.id, CLE.*, element_pathname(element_id) FROM commit_log CL, commit_log_elements CLE WHERE CL.message_id = E'201312271836.rBRIaoHN063395@svn.freebsd.org' AND CLE.commit_log_id = CL.id; id | id | commit_log_id | element_id | revision_name | change_type | element_pathname --------+---------+---------------+------------+---------------+-------------+-------------------------------------------- 500046 | 2269113 | 500046 | 359439 | 337778 | M | /ports/head/sysutils/zfs-periodic/Makefile (1 row) freshports.org=#
OK. Let’s connect into the system_branch_element_revision table and see what references we find:
freshports.org=# SELECT CL.id, CLE.*, element_pathname(CLE.element_id)
FROM commit_log CL, commit_log_elements CLE, system_branch_element_revision SBER
WHERE CL.message_id = E'201312271836.rBRIaoHN063395@svn.freebsd.org'
AND CLE.commit_log_id = CL.id
AND CLE.element_id = SBER.element_id
AND CLE.revision_name = SBER.revision_name;
id | id | commit_log_id | element_id | revision_name | change_type | element_pathname
----+----+---------------+------------+---------------+-------------+------------------
(0 rows)
freshports.org=#
Nothing. Hmmm. That’s bad.
At this point, I start playing with various queries to see how many references I could find to the element_id in question.
freshports.org=# select *
from system_branch_element_revision SBER
WHERE 359439 = SBER.element_id;
system_branch_id | element_id | revision_name
------------------+------------+---------------
1 | 359439 | 1.1
121 | 359439 | 301858
(2 rows)
freshports.org=# select * from commit_log_elements CLE
WHERE 359439 = CLE.element_id;
id | commit_log_id | element_id | revision_name | change_type
---------+---------------+------------+---------------+-------------
2269113 | 500046 | 359439 | 337778 | M
2190381 | 486227 | 359439 | 327772 | M
1611161 | 387853 | 359439 | 1.1 | A
1790229 | 421143 | 359439 | 1.2 | M
1917252 | 443752 | 359439 | 301858 | M
1987472 | 455607 | 359439 | 308502 | M
2018932 | 462046 | 359439 | 312371 | M
(7 rows)
freshports.org=#
There are 7 commits to the Makefile, but only two are recorded in the system_branch_element_revision. That’s not good. We want them all recorded.
Where is it going wrong?
I searched the code for references to system_branch_element_revision and found this stored procedure:
CREATE OR REPLACE FUNCTION ElementTagSet(int4, int4, text) RETURNS boolean AS $$
DECLARE
SystemBranchID ALIAS for $1;
ElementID ALIAS for $2;
RevisionName ALIAS for $3;
ExistingRevisionFound text;
BEGIN
-- has this element already been tagged with this version?
ExistingRevisionFound := RevisionForTagGet(SystemBranchID, ElementID);
IF ExistingRevisionFound IS NULL THEN
insert into system_branch_element_revision
(system_branch_id, element_id, revision_name)
values (SystemBranchID, ElementID, RevisionName);
END IF;
return 1;
END;
$$
LANGUAGE plpgsql;
I also found log output which showed that the example commit executed this SQL (note, the previous queries were on the production database; the following queries are on the test database; thus, the element_id in question may vary):
sql = 'select ElementTagSet(1, 359439, '337778')'
Let’s see what the above stored procedure does with that input.
freshports.org=# begin; BEGIN freshports.org=# select ElementTagSet(1, 359439, '337778'); elementtagset --------------- t (1 row) freshports.org=# select * from system_branch_element_revision where revision_name = '337778'; system_branch_id | element_id | revision_name ------------------+------------+--------------- (0 rows) freshports.org=# rollback; ROLLBACK freshports.org=#
Clearly, that’s not working as designed.
Digging deeper
Let’s see what RevisionForTagGet() returns. This function is straight forward:
CREATE OR REPLACE FUNCTION RevisionForTagGet(int4, int4 ) RETURNS text AS $$
DECLARE
SystemBranchID ALIAS for $1;
ElementID ALIAS for $2;
TempRevisionName text;
-- for a given Element and TagName (System version id)
-- return the revision name assocated with it.
BEGIN
select revision_name
into TempRevisionName
from system_branch_element_revision
where system_branch_id = SystemBranchID
and element_id = ElementID;
return TempRevisionName;
END;
$$ LANGUAGE 'plpgsql';
freshports.org=# select RevisionForTagGet(1, 359439);
revisionfortagget
-------------------
1.1
(1 row)
freshports.org=#
Looking at ElementTagSet() on line 14, I think that’s the key. What we need to do is an insert, and that insert is not happening. I think the cause is the ElementTagSet() function, which needs to do an insert if the tuple is not already in the table. That probably means that RevisionForTagGet() is no longer required. I think the fix is pretty straight forward, but I’ve run out of time today.
