RESTRICTED not correctly reported

May 15th, 2009

The RESTRICTED value on ports has been incorrectly recorded for some time.

The code fix:

$ cvs di port.pm
Index: port.pm
===================================================================
RCS file: /home/repositories/freshports-1/scripts/port.pm,v
retrieving revision 1.54
diff -r1.54 port.pm
168c168
<               $restricted_alt      = FreshPorts::Utilities::NULLIfEmpty($dbh, $this->{restricted_alt});
—
>               $restricted_alt      = FreshPorts::Utilities::NULLIfEmpty($dbh, $this->{restricted});

This is clearly a case of insufficient testing.

I found this bug when wondering why I could not find a package for www/linux-flashplugin9. nox pointed out that it was RESTRICTED: Redistribution not allowed

Eh? That’s not what FreshPorts was telling me. I went looking, ran a few tests, and found the error.

After making the change, I ran a script which refreshes the RESTRICTED field of every port. I then enountered these warnings:

$ perl
set-restricted-no-cdrom.pl > loading.txt
WARNING:  nonstandard use of \\ in a string literal
LINE 1: update ports set restricted = '', no_cdrom = '"no commercial...
                                                     ^
HINT:  Use the escape string syntax for backslashes, e.g., E'\\'.
WARNING:  nonstandard use of \\ in a string literal
LINE 1: update ports set restricted = '"License doesn\\''t allow bin...
                                      ^
HINT:  Use the escape string syntax for backslashes, e.g., E'\\'.
WARNING:  nonstandard use of \\ in a string literal
LINE 1: update ports set restricted = 'Violates djb\\''s license', n...
                                      ^
HINT:  Use the escape string syntax for backslashes, e.g., E'\\'.
WARNING:  nonstandard use of \\ in a string literal
LINE 1: update ports set restricted = '"djb\\''s packaging license d...
                                      ^
HINT:  Use the escape string syntax for backslashes, e.g., E'\\'.
WARNING:  nonstandard use of \\ in a string literal
LINE 1: update ports set restricted = '"Unsure of djb\\''s license"'...
                                      ^
HINT:  Use the escape string syntax for backslashes, e.g., E'\\'.

Those fix-up scripts are pretty old. They may predate the PostgreSQL E’quoting’ mechanism. I should fix them too.

count before fix: 155
count after fix: 390

FYI, this bug has been around since Fri Jun 6 19:13:41 2008 EDT (11 months, 1 week ago).

Oops.

Got DLT?

March 3rd, 2009

I have 7 drives for sale: TZ89N-AV SCSI DLT 7000 tape drive.

See my ebay sale.

Virtual category: docs

February 25th, 2009

The new virtual category docs was recently added.

Interestingly enough, FreshPorts has a /docs/ directory, which directly conflics with this. As such, if you were to browe to http://www.freshports.org/docs/, you’d see this:

Index of /docs

      Name                        Last modified      Size  Description

Parent Directory - 404-for-virtual-pages.php 25-Oct-2007 12:57 8.2K Using 404 errors to serve up virtual pages and virtual categories AddingNewFieldsToPortsTable 25-Oct-2007 12:57 625 CachingStrategy 18-Jul-2006 03:54 2.6K VuXML.php 18-Mar-2007 08:41 29K commits.xml 21-May-2003 01:38 2.1K sample xml file for a commit message from FreeBSD’s cvs-all mailing list commits_2.xml 21-May-2003 01:38 1.4K another samle xml file encodings.php 25-Oct-2007 12:57 1.9K fp-updates.dtd 18-Oct-2007 18:44 3.8K The data definition document for the FreshPorts/FreshSource XML physical_database.gif 11-Nov-2001 02:39 26K FreshPorts/FreshSource database diagram sample_message.1 04-Jan-2001 17:50 2.7K A sample email sample_message.1.xml 21-May-2003 01:38 1.6K A sample email scripts/ 14-Feb-2007 22:22 - Sample scripts for FreshPorts

I have fixed this problem for the BETA website, where http://beta.freshports.org/docs/ where you may view the new docs category.

I’ll fix up the production website once I get http://docs.freshports.org/ running. :)

List of categories

February 25th, 2009

This is news to me (thanks Pav):

$ make -V VALID_CATEGORIES
accessibility afterstep arabic archivers astro audio benchmarks biology cad chinese comms converters databases deskutils devel docs dns editors elisp emulators finance french ftp games geography german gnome gnustep graphics hamradio haskell hebrew hungarian ipv6 irc japanese java kde kld korean lang linux lisp mail math mbone misc multimedia net net-im net-mgmt net-p2p news palm parallel pear perl5 plan9 polish portuguese ports-mgmt print python ruby rubygems russian scheme science security shells spanish sysutils tcl textproc tk ukrainian vietnamese windowmaker www x11 x11-clocks x11-drivers x11-fm x11-fonts x11-servers x11-themes x11-toolkits x11-wm xfce zope

This produces a canonical list of categories. This includes virtual categories as well.

For my own notes: Creating a New Category.

Welcome to Unemployment

February 9th, 2009

Welcome to unemployment.

What was ironic about today: At lunch, I dropped off a cheque to the car dealer for the car I’d picked up two Fridays ago. At 2pm, I was fired. DOH!

Bummer.

My resume.

Thank you.

Caching of non-ports fixed

September 29th, 2008

I think I’ve fixed a problem with caching of non-ports. I was looking for something to do tonight after I could not sleep. I found a old post about a this issue and figured I could fix it.

The fix is in BETA. I’ll monitor it for a while before pushing it to production.

Full diff here.

New (Fancy?) Graphs!

September 23rd, 2008

Some new graphs were put on the beta site last night.  They use a combination of jQuery and Flot to generate the graphs.  All the backend code is php (but is quite ugly and small).  The real meat of the important code is in the javascript.  I’ve written an explanation of how the code works for those who are interested.  There are not a lot of graphs right now because I just wanted to get a POC out there before I work on adding more graphs.  If Dan can come up with an SQL query to get the data I’m fairly certain I can graph it and may even be able to add fancy things like selections and zooming.  So if you have any suggestions for what to graph please mail me about them.

The only downside of this approach is it requires a modern browser running javascript.  I believe there are problems with Konqueror under KDE 3 and IE6.  I think the support hinges upon the canvas element, but don’t quote me on that.  It renders fine on the handful of browsers I use so if it doens’t work for you I think it’s time to update to a newer version of your browser of choice.

Ports freeze

September 17th, 2008

Vladimir Chukharev wrote in to mention a problem with the ports freeze. FreshPorts usually puts up a notice indicating a Ports Freeze is in effect when the freeze is inacted. This freeze is effected by making a special change to CVSROOT/approvers.

FreshPorts looks for this change and sets a flag. The website notices this flag and adds a notice to every webpage.

It took me a couple of hours to find and fix the problem. First, I had to remember how this processing is done. Eventually, the problem came down to this one line:

if ($File eq $FreshPorts::Constants::CVSROOT_Ports_Approvers) {

The problem is the value for that constant. It is ‘CVSROOT-ports/approvers’. The commit in question affects CVSROOT/approvers. However, the actual file in the repository is CVSROOT-ports/approvers. FreshPorts needs to know this when fetching the file from the repository via http. Previously, FreshPorts invoked a special function to convert from the commit file name to the repository file name. At present, it looks like this:

        # before we do anything, convert the FileName appropriately
        #
#       $FilePath = ConvertFilePath($FilePath);

Yes, it is commented out. This function call was originally added on Tue Oct 16 19:55:51 2007 EDT (11 months ago). Then, on Fri Nov 9 02:38:39 2007 EST (10 months, 1 week ago), I see this cvs log comment:

I now think ConvertFilePath is wrong.  I should not munge the file name like
that.  Do it at fetch time!

Now I think I’m wrong again. But only on the conditional above. I changed it to this:

if ($File eq $FreshPorts::Constants::CVSROOT_Approvers && $Repository eq $FreshPorts::Constants::Repository_Ports) {

This constant is the un-munged filename as found in the commit message. I now explicity check to see what repository we are working against. Even this may prove problematic as FreeBSD continues moviing to svn from cvs.

There are more details to the fix, and you are welcome to review the whole diff if you like.

Me, I’m just off to sleep. I’m dead.

svn versus cvs emails

September 4th, 2008

FreshPorts processes the cvs commit logs from the FreeBSD cvs-all mailing list. It parses these emails into XML. FreeBSD is in the process of moving to svn.

This email will demonstrate of the differences between the two email formats and highlight any issues that might surround the move to processing svn instead of cvs logs.

CVS emails

The following is a cvs email for a recent commit.

Message-Id: <200809050207.m8527Tu1007048@repoman.freebsd.org>
X-Authentication-Warning: repoman.freebsd.org: svn2cvs set sender to davidxu@repoman.freebsd.org using -f
From: David Xu <davidxu@FreeBSD.org>
Date: Fri, 5 Sep 2008 02:07:25 +0000 (UTC)
To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org
Subject: cvs commit: src/sys/kern uipc_mqueue.c
X-FreeBSD-CVS-Branch: HEAD

davidxu     2008-09-05 02:07:25 UTC

  FreeBSD src repository

  Modified files:
    sys/kern             uipc_mqueue.c
  Log:
  SVN rev 182776 on 2008-09-05 02:07:25Z by davidxu

  Fix lock name conflict.

  PR:     kern/127040

  Revision  Changes    Path
  1.32      +1 -1      src/sys/kern/uipc_mqueue.c

SVN emails

Message-Id: <200809050207.m8527Pod090353@svn.freebsd.org>
From: David Xu <davidxu@FreeBSD.org>
Date: Fri, 5 Sep 2008 02:07:25 +0000 (UTC)
To: src-committers@freebsd.org
Subject: svn commit: r182776 - head/sys/kern
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Author: davidxu
Date: Fri Sep  5 02:07:25 2008
New Revision: 182776
URL: http://svn.freebsd.org/changeset/base/182776

Log:
  Fix lock name conflict.

  PR:	kern/127040

Modified:
  head/sys/kern/uipc_mqueue.c

Modified: head/sys/kern/uipc_mqueue.c
==============================================================================
--- head/sys/kern/uipc_mqueue.c	Fri Sep  5 01:36:30 2008	(r182775)
+++ head/sys/kern/uipc_mqueue.c	Fri Sep  5 02:07:25 2008	(r182776)
@@ -1499,7 +1499,7 @@ mqueue_alloc(const struct mq_attr *attr)
 		mq->mq_maxmsg = default_maxmsg;
 		mq->mq_msgsize = default_msgsize;
 	}
-	mtx_init(&mq->mq_mutex, "mqueue", NULL, MTX_DEF);
+	mtx_init(&mq->mq_mutex, "mqueue lock", NULL, MTX_DEF);
 	knlist_init(&mq->mq_rsel.si_note, &mq->mq_mutex, NULL, NULL, NULL);
 	knlist_init(&mq->mq_wsel.si_note, &mq->mq_mutex, NULL, NULL, NULL);
 	atomic_add_int(&curmq, 1);

XML

Both emails must produce the same XML:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE UPDATES PUBLIC "-//FreshPorts//DTD FreshPorts 2.0//EN" "http://www.freshports.org/docs/fp-updates.dtd">

<UPDATES Version="1.3.2.1">
    <UPDATE>
        <DATE Year="2008" Month="9" Day="5"></DATE>
        <TIME Timezone="UTC" Hour="2" Second="25" Minute="7"></TIME>
        <OS Id="FreeBSD" Branch="HEAD"></OS>
        <LOG>SVN rev 182776 on 2008-09-05 02:07:25Z by davidxu

Fix lock name conflict.

PR:     kern/127040</LOG>
        <PEOPLE>
            <UPDATER Handle="davidxu"></UPDATER>
        </PEOPLE>
        <MESSAGE EncodingLosses="false" Subject="cvs commit: src/sys/kern uipc_mqueue.c" Id="200809050207.m8527Tu1007048@repoman.freebsd.org">
            <DATE Year="2008" Month="9" Day="5"></DATE>
            <TIME Timezone="UTC" Hour="2" Second="25" Minute="7"></TIME>
            <REPOSITORY>src</REPOSITORY>
            <TO Email="src-committers@FreeBSD.org"></TO>
            <TO Email="cvs-src@FreeBSD.org"></TO>
            <TO Email="cvs-all@FreeBSD.org"></TO>
        </MESSAGE>
        <FILES>
            <FILE Revision="1.32" Action="Modify" Path="src/sys/kern/uipc_mqueue.c" Changes="+1 -1"></FILE>
        </FILES>
        <PR Id="kern/127040"></PR>
    </UPDATE>
</UPDATES>

Analysis will reveal what, if any, information is missing from the svn commit email.

Other issues

FreshPorts avoids processing duplicate commits by using the message ID as a unique identifier. We must consider that the SVN and the CVS commit emails use different message IDs for the same commit. Therefore we must be careful not to proces s both CVS and SVN email for the same commit.

We may be able to do this via procmail.

Copying database dumps around

August 30th, 2008

One issue which has interested me for a while is moving databases from one server to another. My traditional approach involves taking down the website, dumping the database, copying the dump file, loading it up, and then changing the DNS to point to the new server.

With the goal of minimizing downtime, I’ve been considering this approach:

  1. dump the database
  2. scp the dump to the new server
  3. turn off updates on the old server
  4. dump again
  5. rsync to the new server
  6. load the dump onto the new server
  7. change the DNS

The goal is to reduce the time between turning off updates and changing the DNS. I have previous timings for dump and scp times. Let’s now look at rsync times. We will also trying dumping without compression.

[dan@nyi:~/transfer] $ time pg_dump freshports.org > freshports.sql

real    26m12.041s
user    0m17.144s
sys     0m19.165s
[dan@nyi:~/transfer] $

Now, when I do another rsync, after the dump, it takes just 10 minutes to rsync:

 $ time /usr/local/bin/rsync -e "ssh -i /home/dan/.ssh/supernews-maildir-rsync-key" --recursive --delete -avz --stats --progress dan@nyi:transfer ~/
receiving file list ...
2 files to consider
transfer/freshports.sql
  2832187798 100%    4.50MB/s    0:10:00 (xfer#1, to-check=0/2)

Number of files: 2
Number of files transferred: 1
Total file size: 2832187798 bytes
Total transferred file size: 2832187798 bytes
Literal data: 95236822 bytes
Matched data: 2736950976 bytes
File list size: 97
File list generation time: 0.011 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 425578
Total bytes received: 21865883

sent 425578 bytes  received 21865883 bytes  36099.53 bytes/sec
total size is 2832187798  speedup is 127.05

real    10m16.993s
user    0m25.544s
sys     0m11.944s
[dan@supernews:~/bin] $

Now let’s try a gzip approach.


 $ time pg_dump freshports.org | gzip >  freshports.sql.gz
zreverse-i-search)`gzip': time pg_dump freshports.org | gzip >  freshports.sql.gz

real    23m45.288s
user    5m42.822s
sys     0m15.884s
[dan@nyi:~/transfer] $

Initial reports indicate dumping to gzip is faster than no gzip.

Now trying an rsync of a gzip file… It’s not looking promising.

transfer/freshports.sql.gz
   552743046 100%  222.41kB/s    0:40:26 (xfer#1, to-check=0/2)

Number of files: 2
Number of files transferred: 1
Total file size: 552743046 bytes
Total transferred file size: 552743046 bytes
Literal data: 552014422 bytes
Matched data: 728624 bytes
File list size: 92
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 164646
Total bytes received: 552200103

sent 164646 bytes  received 552200103 bytes  227076.98 bytes/sec
total size is 552743046  speedup is 1.00

real    40m31.289s
user    0m16.055s
sys     0m7.961s

Well, that takes forty minute. it seems better to rsync raw text.