|
|
September 11th, 2007
After my overnight security report audit came in, I noticed that Apache needed to be upgraded. I went to FreshPorts to see if a fix had been committed. While there, I noticed a lack of vuxml skulls against the latest versions of Apache. Checking the BETA website, I saw it was correctly marked. More digging found the problem. In the process, I improved some error reporting in the scripts so that this problem should be brought to my attention much sooner.
Things should be back to normal now.
Posted in Bug fixes, vuxml | No Comments »
August 25th, 2007
As reported by sem@, there is a problem with the display of master/slave relationships within FreshPorts. The relationship is stored in the ports table, with the master_port attribute being a pointer to the master port. This text field typically has values such as this:
freshports.org=# SELECT DISTINCT(master_port)
freshports.org-# FROM ports_active
freshports.org-# where master_port <> ”
freshports.org-# AND master_port not like ‘/usr/home/dan/%’
freshports.org-# ORDER BY master_port
freshports.org-# LIMIT 10;
master_port
————————–
archivers/unrar
audio/aylet
audio/festalon
audio/festvox-us1-mbrola
audio/gbsplay
audio/gqmpeg
audio/mbrola
audio/napster
audio/scrobbler
audio/timidity++
(10 rows)
freshports.org=#
The query to find any slave ports of a given port is:
SELECT id AS slave_port_id,
name AS slave_port_name,
category_id AS slave_category_id,
category AS slave_category_name
FROM ports_active
WHERE master_port = 'sysutils/bacula-server'
ORDER BY slave_category_name, slave_port_name
Which, on my development server at home, for sysutils/bacula-server gives:
slave_port_id | slave_port_name | slave_category_id | slave_category_name
---------------+-----------------+-------------------+---------------------
13986 | bacula-client | 20 | sysutils
(1 row)
The problem is: bacula-server is not listing bacula-client as a slave port. For that matter, in production, bacula-client isn’t listing bacula-server as its master port either. Strange. Very strange. It must be something very simple that has gone wrong here. This has worked in the past.
FreshPorts determines the MASTERPORT through make -V magic. Something like this:
[dan@ngaio:~/ports/sysutils/bacula-client] $ make -V MASTERPORT \
PORTSDIR=~/ports LOCALBASE=/nonexistentlocal X11BASE=/nonexistentx
sysutils/bacula-server
[dan@ngaio:~/ports/sysutils/bacula-client] $
This magic is accomplished with this patch:
$ less ~/bin/bsd.port.mk.master-slave-patch
--- bsd.port.mk 10 Jun 2004 07:30:19 -0000 1.491
+++ bsd.port.mk 22 Jun 2004 13:48:33 -0000
@@ -913,6 +913,16 @@
MASTERDIR?= ${.CURDIR}
+# Try to determine if we are a slave port. These variables are used by
+# FreshPorts and portsmon, but not yet by the ports framework itself.
+.if ${MASTERDIR} != ${.CURDIR}
+IS_SLAVE_PORT?= yes
+MASTERPORT?= ${MASTERDIR:C/[^\/]+\/\.\.\///:C/[^\/]+\/\.\.\///:C/^.*\/([^\/]+\/[^\/]+)$/\1/}
+.else
+IS_SLAVE_PORT?= no
+MASTERPORT?=
+.endif
+
# If they exist, include Makefile.inc, then architecture/operating
# system specific Makefiles, then local Makefile.local.
So something, somewhere has allowed this information to be wrong. A few days ago, I did notice that my cvsup crontab was going wrong and was not correctly patching the file after the upgrade. Perhaps that was it.
Running a few commits manually shows that is the problem. An unpatched Mk/*.
I’ll work on getting that fixed and then run a script to check for a master port for each port.
Posted in Bug fixes, Master/Slave | 2 Comments »
August 3rd, 2007
This is a starting introduction to the FreshPorts database. Hopefully it will prompt questions. Ask.
The key data component of FreshPorts is the commit. These are stored in the commit_log table. Each commit affects one or more files, known in the database as elements.
The elements table is a self-referencing table and represents the files and directories of the source repository. Here are a few entries from that table:
freshports.org=# select *, element_pathname(id) from element order by id limit 10;
id | name | parent_id | directory_file_flag | status | element_pathname
----+----------+-----------+---------------------+--------+--------------------------------
1 | ports | | D | A | /ports
2 | editors | 1 | D | A | /ports/editors
3 | yudit | 2 | D | A | /ports/editors/yudit
4 | Makefile | 3 | F | A | /ports/editors/yudit/Makefile
5 | pkg | 3 | D | A | /ports/editors/yudit/pkg
6 | DESCR | 5 | F | D | /ports/editors/yudit/pkg/DESCR
7 | files | 3 | D | A | /ports/editors/yudit/files
8 | md5 | 7 | F | D | /ports/editors/yudit/files/md5
9 | www | 1 | D | A | /ports/www
10 | quanta | 9 | D | D | /ports/www/quanta
(10 rows)
freshports.org=#
To relate a commit to the files it touches, the commit_log_ports table is used.
Ports are just a special case of abstraction. The Ports table contains data taken from each Makefile for the port in question. For this, “make -V” is used.
Ports are also elements. Using the element ids from the above query, here are the ports that pertain to those elements:
The key data component of FreshPorts is the commit. These are stored in the commit_log table. Each commit affects one or more files, known in the database as elements.
The elements table is a self-referencing table and represents the files and directories of the source repository. Here are a few entries from that table:
freshports.org=# select id, element_id, category_id, short_description from ports
where element_id between 1 and 10;
id | element_id | category_id | short_description
----+------------+-------------+----------------------------------------------------
1 | 3 | 1 | Multi-lingual unicode text editor with TTF support
2 | 10 | 2 | Comprehensive html/website development environment
(2 rows)
freshports.org=#
Some commits affect ports. This relationship is maintained in the commit_log_ports table.
The categories table is what you think it is:
freshports.org=# select id, element_id, name from categories order by name;
id | element_id | name
-----+------------+---------------
85 | 171607 | accessibility
64 | | afterstep
88 | 159346 | arabic
23 | 350 | archivers
26 | 410 | astro
25 | 386 | audio
42 | 2710 | benchmarks
36 | 869 | biology
35 | 830 | cad
39 | 1660 | chinese
41 | 2191 | comms
27 | 423 | converters
32 | 582 | databases
33 | 802 | deskutils
10 | 84 | devel
84 | 148762 | dns
1 | 2 | editors
63 | | elisp
22 | 245 | emulators
54 | 118514 | finance
47 | 16545 | french
13 | 140 | ftp
3 | 18 | games
118 | | geography
44 | 3747 | german
58 | | gnome
101 | | gnustep
4 | 29 | graphics
96 | | hamradio
77 | | haskell
46 | 11329 | hebrew
51 | 118517 | hungarian
62 | | ipv6
6 | 39 | irc
12 | 129 | japanese
34 | 815 | java
55 | | kde
117 | | kld
37 | 1109 | korean
15 | 171 | lang
66 | | linux
90 | | lisp
19 | 201 | mail
16 | 176 | math
45 | 6412 | mbone
7 | 42 | misc
52 | 118520 | multimedia
8 | 50 | net
95 | 229588 | net-im
92 | 173566 | net-mgmt
98 | 236506 | net-p2p
17 | 179 | news
76 | | offix
40 | 2143 | palm
93 | | paralell
68 | | parallel
89 | | pear
94 | | perl
59 | | perl5
87 | | php
50 | 58316 | picobsd
69 | | plan9
82 | 148764 | polish
111 | 265340 | ports-mgmt
53 | 118523 | portuguese
24 | 360 | print
57 | | python
74 | | ruby
97 | | rubygems
31 | 577 | russian
83 | | scheme
48 | 56065 | science
5 | 34 | security
29 | 465 | shells
100 | | spanish
20 | 218 | sysutils
70 | | tcl80
72 | | tcl81
71 | | tcl82
60 | | tcl83
80 | | tcl84
18 | 188 | textproc
78 | | tk42
73 | | tk80
61 | | tk82
65 | | tk83
79 | | tk84
75 | | tkstep80
49 | 57265 | ukrainian
11 | 94 | vietnamese
56 | | windowmaker
2 | 9 | www
21 | 231 | x11
28 | 428 | x11-clocks
115 | 278073 | x11-drivers
30 | 516 | x11-fm
38 | 1229 | x11-fonts
43 | 3321 | x11-servers
91 | 171611 | x11-themes
9 | 55 | x11-toolkits
14 | 147 | x11-wm
81 | | xfce
67 | | zope
(103 rows)
freshports.org=
You will notice that categories, like ports, have an element_id.
One of the key features of FreshPorts is the notification. Users select what they wish to monitor and add it to their watch list. A watch list consists of a watch_list_id and an element_id.
freshports.org=# select * from watch_list_element limit 10;
watch_list_id | element_id
---------------+------------
4000 | 916
4334 | 112921
4334 | 90346
4334 | 88170
4334 | 101671
3105 | 57670
3105 | 13030
3105 | 68999
3105 | 2487
3105 | 68994
(10 rows)
freshports.org=#
In turn, each watch list is owned by a user. Users can have multiple watch lists:
freshports.org=# select id, user_id, name from watch_list order by user_id limit 10;
id | user_id | name
-------+---------+--------
10276 | 1 | stuff
10275 | 1 | things
8609 | 2 | main
10277 | 2 | other
1247 | 3 | main
1248 | 4 | main
1249 | 5 | main
1250 | 6 | main
1251 | 7 | main
4 | 9 | main
(10 rows)
freshports.org=#
Posted in Development, Watch Lists | No Comments »
July 31st, 2007
The repo is now online: See http://cvs.freshports.org/cgi-bin/cvsweb.cgi/.
You can also get to a cvs server at the same hostname.
Posted in Announcements | 4 Comments »
July 28th, 2007
I am just back from OSCON.
While there, I attempted to incorporate some CSS patches into FreshPorts…. I failed.
It is time to Open Source FreshPorts. It uses PHP (website), Perl (message processing), and Python (cache). It’ll take me a while to get the infrastructure into place. I have servers suitable for this purpose. I just need the time. :)
Posted in Advocacy, Announcements, Development, New ideas | 3 Comments »
July 24th, 2007
For more detail, read Duplicate file in cvs-all commit causes problems.
[dan@ngaio:/usr/websites/freshports.org/scripts] $ cvs di process_cvs_mail.pl
Index: process_cvs_mail.pl
===================================================================
RCS file: /home/repositories/freshports-1/scripts/process_cvs_mail.pl,v
retrieving revision 1.10
diff -r1.10 process_cvs_mail.pl
70c70
< FreshPorts::Utilities::ReportError('err', "No message ID found for this commit message (" . $Message_Subject . ").\n\nIs this a corrupted commit or email?", 1)
---
> FreshPorts::Utilities::ReportErrorEmailNoPrint(’err’, “No message ID found for this commit message (” . $Message_Subject . “).\n\nIs this a corrupted commit or email?”, 1)
342c342
< FreshPorts::Utilities::ReportError('err', "Duplicate file name ('$path') found in commit message (" . GetMessage_Id($message) . ").\n\nIs this a corrupted commit or email?", 1)
---
> FreshPorts::Utilities::ReportErrorEmailNoPrint(’err’, “Duplicate file name (’$path’) found in commit message (” . GetMessage_Id($message) . “).\n\nIs this a corrupted commit or email?”, 0)
393c393
< FreshPorts::Utilities::ReportError('err', "Duplicate file name ('$path') found in commit message (" . GetMessage_Id($message) . ").\n\nIs this a corrupted commit or email?", 1)
---
> FreshPorts::Utilities::ReportErrorEmailNoPrint(’err’, “Duplicate file name (’$path’) found in commit message (” . GetMessage_Id($message) . “).\n\nIs this a corrupted commit or email?”, 0)
$
Posted in Development | No Comments »
July 24th, 2007
For more detail, read Duplicate file in cvs-all commit causes problems.
$ cvs di -u utilities.pm
Index: utilities.pm
===================================================================
RCS file: /home/repositories/freshports-1/scripts/utilities.pm,v
retrieving revision 1.14
diff -u -r1.14 utilities.pm
--- utilities.pm 17 Mar 2007 13:49:42 -0000 1.14
+++ utilities.pm 27 Jun 2007 02:27:26 -0000
@@ -128,7 +128,7 @@
my $email = $die;
- _ReportErrorHelper($level, $message, $email, $die);
+ _ReportErrorHelper($level, $message, $email, $die, 1);
}
sub ReportErrorEmail($;$;$;$) {
@@ -137,19 +137,31 @@
my $email = shift;
my $die = shift;
- _ReportErrorHelper($level, $message, $email, $die);
+ _ReportErrorHelper($level, $message, $email, $die, 1);
}
-sub _ReportErrorHelper($;$;$;$) {
+sub ReportErrorEmailNoPrint($;$;$;$) {
my $level = shift;
my $message = shift;
my $email = shift;
my $die = shift;
+ _ReportErrorHelper($level, $message, $email, $die, 0);
+}
+
+sub _ReportErrorHelper($;$;$;$;$) {
+ my $level = shift;
+ my $message = shift;
+ my $email = shift;
+ my $die = shift;
+ my $print = shift;
+
my $suffix = $FreshPorts::Config::scriptpath;
Sys::Syslog::syslog($level, $message . " ($suffix)");
- print $message . "\n";
+ if ($print) {
+ print $message . "\n";
+ }
if ($email) {
SendEmailNotice($FreshPorts::Config::SystemOwnerEmail, $message);
$
Posted in Development | No Comments »
July 22nd, 2007
In case you hadn’t noticed, FreshPorts is now running on the new server. With hardware donated by iXsystems and 3Ware, and colocation/bandwidth provided by SuperNews, the box has been installed for some time. Blame me for the delayed transition. BSDCan, PGCon, moving to Florida for a new job, getting laid off due to outsourcing, and trying to find a new job have taken priority lately.
More reading about this server:
Dual Opteron server 15 Jun 2006
Dual Opteron Server - the drives 22 Jun 2006
More dual opteron images 9 Jul 2006
Dual Opteron - Raptor Installation 10 Aug 2006
CLI for 3Ware 9550SX-8LP 14 Aug 2006
NetSaint plugin for 3Ware RAID card 18 Aug 2006
Dual Opteron - better SATA cables for the Raptors 19 Aug 2006
3Ware - Manage your RAID arrays via http 28 Aug 2006
Posted in Hardware, Server | No Comments »
July 6th, 2007
Søren Straarup brought this up today. FreshSource does not link to the right place in CVSWEB for commits to CVSROOT. For example, look at this commit. If you click on the diff link, it goes to the wrong place. It goes to
http://www.freebsd.org/cgi/cvsweb.cgi/CVSROOT/modules.diff?r1=1.18319&r2=1.18318
instead of
http://www.freebsd.org/cgi/cvsweb.cgi/CVSROOT-ports/modules.diff?r1=1.18319&r2=1.18318
After discussions with Simon Nielsen and is Søren Straarup, we found a few fine examples:
The main issue, as I see it now, is to associate the given commit with a particular repository. In the beginning, FreshPorts processed only commits from the ports repository. Then, it started to process all commits, but did not differentiate between the various repository.
The problem is compounted by the cvs-all emails which do not differentiate between the various CVSROOT files. The full path is one most file names, but not on CVSROOT.
What we may be able to do is process the line “FreeBSD .* repository” and use that. It would require a change to the XML template, and some significant processing changes within FreshPorts to grab that value.
This is where the real thinking starts. :) This seems like an AWFUL lot of work just to get the path to the CVSROOT file correct. Perhaps I should just amend the path of such files as they go into the XML and not mess around with the XML otherwise.
What do you suggest?
Posted in Bug fixes, Development, FreeBSD, New ideas | 1 Comment »
June 26th, 2007
Greetings from southern Florida. If you haven’t heard, I moved to Jupiter, Florida. Yes, PGCon and BSDCan will be in Ottawa in 2008. My relocation does not affect those events.
It was this commit to print/gutenprint that caused the problem. You will notice that one file is touched twice by that commit. It is best demonstrated by the last two lines of the log message:
1.32 +1 -8 ports/print/gutenprint/Makefile
1.33 +0 -0 ports/print/gutenprint/Makefile
FreshPorts was long ago coded to reject any such messages. The code in question is:
if (defined($TrackDuplicates{$path})) {
FreshPorts::Utilities::ReportError('err',
"Duplicate file name ('$path') found in commit message (" .
GetMessage_Id($message) . ").\n\nIs this a corrupted commit or email?", 1)
} else {
$TrackDuplicates{$path} = 1;
}
This code uses a hash/array with the file pathname as the key/index. It also invokes ReportError() with a final parameter of 1, which means fatal error.
I decided this wasn’t a fatal error any more. I changed the 1 to a 0 and tried processing the commit again. I started thinking there was an error somewhere else in the code, because I was getting this error:
syntax error at line 1, column 0, byte 0 at /usr/local/lib/perl5/site_perl/5.8.8/mach/XML/Parser.pm line 187
I started playing around with the XML file in case duplicate entries were a problem there. It was then that I noticed the problem (but not immediately). The XML file contained non-xml. At the top of the file was:
Duplicate file name ('ports/print/gutenprint/Makefile') found in commit message (200706261507.l5QF7lEE060586@repoman.freebsd.org).
Is this a corrupted commit or email?
OK, I’m printing out error messages and they are being piped to the file. I know a simple way around that. Don’t print, email. There is already a function that will email me the error message as well as log it via syslog. I’ll use that instead. Problem is, that prints too. So I crreated a new function and used it. This diff shows what I did:
utilities patch
This creates a new function, ReportErrorEmailNoPrint(), and adds a new parameter to _ReportErrorHelper(). This allows for the conditional printing of the error message.
The XML processing code changed to:
process_cvs_mail.pl patch
Tested. Fixed. Done. And soon to be in production so that commit message can be processed.
Posted in Bug fixes, Development | No Comments »
|
