Listening daemon out of date

Back in 2007, perhaps earlier, I wrote a Python script which listens for various PostgreSQL events. In particular, this script will invalidate the cache for a port when it is updated. There are several other such tasks, such as the creation of a new category.

When the commit processing daemon (freshports) detects that a new category has been created, an event (listen_category_new) is triggered. When the listening daemon (fp-listen) sees that notice, it runs the script ProcessCategoryNew().

Upon startup, the fp-listen daemon will read the contents of the listen_for table to determine the list of events to watch.

ProcessCategoryNew() will fetch the URL: http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/www/en/ports/categories?rev=HEAD;content-type=text%2Fplain

That’s CVS. We haven’t used CVS in some time. This is clearly out of date and needs to be updated. The URL still works, but the last time it was updated was March 2011. The new URL is http://svnweb.freebsd.org/doc/head/en_US.ISO8859-1/htdocs/ports/categories?revision=HEAD.

Looking at that URL, the last category added is enlightenment, back on 26 Feb 2009. That predates the change to svn. Therefore, FreshPorts has not missed anything. Yet.

I looked around for code which used listen_category_new. I found nothing. I looked in the database setup. Nothing.

I tried grep:

$ ': grep -i 'new category' *
verifyport.pm: print "creating new category $category_name\n";
verifyport.pm: FreshPorts::Utilities::ReportError('warning', "creating new category $category_name", 0);
verifyport.pm: FreshPorts::Utilities::ReportError('warning', "failed to create new category $category_name", 1);

I started looking in verifyport.pm and found this code:

# we need to create this catgory.
# remember to grab ports/<category>/pkg/COMMENT
print "creating new category $category_name\n";
FreshPorts::Utilities::ReportError('warning', "creating new category $category_name", 0);

$category->{is_primary} = 1;
$category_id = $category->save();
if (!defined($category_id)) {
    FreshPorts::Utilities::ReportError('warning', "failed to create new category $category_name", 1);
}

That reference to ports//pkg/COMMENT is outdated. The category description now resides in ports//Makefile as a COMMENT entry.

Further investigation showed that the code is doing the right thing and pulling the description from that location.

But what about virtual categories? They will still be in htdocs/ports/categories

The answer is no, FreshPorts is not pulling the description for virtual categories. Look at the current list of categories. You will find 5 entries with a description of ‘no description supplied’.

I kept digging. It’s quite wonderful how much you forget about code.

I grep’d for ports/categories and found:

$ grep ports/categories *
constants.pm:$FreshPorts::Constants::Categories                                         = 'www/en/ports/categories';

You’ll notice that file is wrong. It should be: doc/head/en_US.ISO8859-1/htdocs/ports/categories I think.

htdocs/ports/categories

Ahh, so where is that used?

$ grep ::Categories *
constants.pm:$FreshPorts::Constants::Categories                                         = 'www/en/ports/categories';
special_processing_files.pm:    if ($File eq $FreshPorts::Constants::Categories) {
special_processing_files.pm:            my $SRCDIR  = dirname($FreshPorts::Constants::Categories);
special_processing_files.pm:            my $FILE    = basename($FreshPorts::Constants::Categories);

Yes, I remember special processing. Every file update in every commit invokes that script, looking for $FreshPorts::Constants::Categories. If found, it sets a flag: $FreshPorts::Config::WWWENPortsCategoriesFlag

That flag is picked up by the freshports daemon (fp-daemon) and used to invoke the job-waiting.pl script.

In turn, process_www_en_ports_categories.sh is invoked. In turn categories_update_descriptions.pl is invoked and the descriptions are updated. All of them.