Fixing the category creation code

Leave a Comment / By /

Following on from yesterday’s blog post about the new filesystems category, today I hope to fix the code and get the website up to date again.

At present, the code is spinning, trying to fetch category information using outdate code which assumes subversion is still in use.

[16:46 dvl-ingress01 dvl ~freshports/message-queues/spooling] % ps auwwx | grep perl
freshports 47970  0.0  0.0 57968 43284  -  IJ   10:51   0:01.18 /usr/local/bin/perl /usr/local/libexec/freshports/load_xml_into_db.pl
/var/db/ingress/message-queues/incoming/2024.11.07.10.49.25.000000.43862c0fcfc10374266bf69552e002212a3061a1.xml
dvl        10467  0.0  0.0  6664  2172  1  R+J  16:47   0:00.00 grep perl

Looking at the log file:

[16:47 dvl-ingress01 dvl ~freshports/message-queues/spooling] % ls -l
total 25
-rw-r--r--  1 freshports freshports      0 2024.11.07 10:51 2024.11.07.10.49.25.000000.43862c0fcfc10374266bf69552e002212a3061a1.errors
-rw-r--r--  1 freshports freshports 107417 2024.11.07 16:47 2024.11.07.10.49.25.000000.43862c0fcfc10374266bf69552e002212a3061a1.log

[16:48 dvl-ingress01 dvl ~freshports/message-queues/spooling] % tail -20 2024.11.07.10.49.25.000000.43862c0fcfc10374266bf69552e002212a3061a1.log


num of params = 8
error invoking script /usr/local/libexec/freshports/svn-up-file.sh : usage /usr/local/libexec/freshports/svn-up-file.sh SVNDIR SVNITEM REVISION (e.g. /usr/local/libexec/freshports/svn-up-file.sh /usr/ports sysutils/bacula-server 1234)


sleeping after fetch failed for (/usr/ports/filesystems filesystems Makefile)
fetch failed, sleeping...
about to fetch = 'sh /usr/local/libexec/freshports/svn-up-file.sh  REPODIR SVNITEM REVISION /usr/ports/filesystems filesystems Makefile head '''
fetch result = 256
that fetch failed.  What do to?


num of params = 8
error invoking script /usr/local/libexec/freshports/svn-up-file.sh : usage /usr/local/libexec/freshports/svn-up-file.sh SVNDIR SVNITEM REVISION (e.g. /usr/local/libexec/freshports/svn-up-file.sh /usr/ports sysutils/bacula-server 1234)


sleeping after fetch failed for (/usr/ports/filesystems filesystems Makefile)
fetch failed, sleeping...
about to fetch = 'sh /usr/local/libexec/freshports/svn-up-file.sh  REPODIR SVNITEM REVISION /usr/ports/filesystems filesystems Makefile head '''
[16:48 dvl-ingress01 dvl ~freshports/message-queues/spooling] %

What’s in this svn-up-file.sh?

[16:49 dvl-ingress01 dvl ~/scripts] % cat svn-up-file.sh
#!/bin/sh
#
# $Id: svn-up-file.sh,v 1.1 2012-08-15 11:49:10 dan Exp $
#
# Copyright (c) 1999-2012 DVL Software
#
# This script used to svn up files...

echo "num of params = $#"
if  [ $# -ne 3 ];
then echo error invoking script $0 : usage $0 SVNDIR SVNITEM REVISION \(e.g. $0 /usr/ports sysutils/bacula-server 1234\)
  exit 1
else
    SVNDIR=$1
    SVNITEM=$2
    REVISION=$3

    # we may not need this cd...
    echo cd to ${SVNDIR}
    cd ${SVNDIR}
    
    pwd
    echo "doing this: svn up -r ${REVISION} ${SVNITEM}"
    svn up -r ${REVISION} ${SVNITEM}
    exit $?
fi

It’s trying to full down the Makefile for the filesystems category. That’s how it used to work. However, with git, that file is already here.

[16:49 dvl-ingress01 dvl ~/scripts] % ls -l /jails/freshports/usr/ports/filesystems/Makefile 
-rw-r--r--  1 root wheel 3493 2024.11.07 10:45 /jails/freshports/usr/ports/filesystems/Makefile
[16:50 dvl-ingress01 dvl ~/scripts] % head /jails/freshports/usr/ports/filesystems/Makefile
    COMMENT = File systems and related utilities

    SUBDIR += acfgfs
    SUBDIR += afuse
    SUBDIR += archivemount
    SUBDIR += automount
    SUBDIR += avfs
    SUBDIR += bindfs
    SUBDIR += cfs
    SUBDIR += chironfs
[16:51 dvl-ingress01 dvl ~/scripts] %

Eventually, the code will pull that COMMENT out of the file and use it when creating the category within the FreshPorts database.

Tracking down the code

modules/svn-up-file.sh is invoked from modules/utilities.php, within the FetchFileURL() file. In turn, that is called from FetchFile() within the same file. FetchFile() is only called from within modules/category.pm

I think that code can be fixed by this:

[16:56 dvl-ingress01 dvl ~/modules] % svn di category.pm
Index: category.pm
===================================================================
--- category.pm	(revision 6098)
+++ category.pm	(working copy)
@@ -123,7 +123,7 @@
 }
 
 sub FetchByID {
-	my $this	= shift;
+	my $this = shift;
 
 	my $dbh;
 	my $sql;
@@ -272,14 +272,8 @@
 	print "SRCDIR =$SRCDIR\n";
 	print "FILE   =$FILE\n";
 
-	if (FreshPorts::Utilities::FetchFile($DESTDIR, $SRCDIR, $FILE, $FreshPorts::Constants::HEAD)) {
-		$description = _description_read($CommitBranch, $category);
+	$description = _description_read($CommitBranch, $category);
 
-	} else {
-		FreshPorts::Utilities::ReportError('warning', "Could not fetch file for '$DESTDIR' '$SRCDIR' '$FILE'.  Error code = " . ($? >> 8), 0);
-		$description = 'No description supplied (pkg/COMMENT not found)';
-	}
-
 	# get rid of the trailing CR/LF.
 	chomp $description;

Getting the description

I wanted to look closer at _description_read():

sub _description_read {
        my $Repository   = shift;
        my $CommitBranch = shift;
        my $category     = shift;

        my $description = '';

        my $TmpFile = FreshPorts::Utilities::TmpFileName("$category.make-error");

        my $ErrorMessage = '';  # stores the result of the latest make command
                                # in case we need it for error reporting
        my $OtherErrors  = '';  # gets the results of the TmpFile used to collect errors.

        # with subversion, FreshPorts held a different working copy of the repo for the quarterly branch
        # and another for head. 
        my $REPODIR_CHROOT = $FreshPorts::Config::PortsDir;

        my $makecommand = "/usr/local/bin/sudo /usr/sbin/jexec $FreshPorts::Config::JailName $REPODIR_CHROOT $FreshPorts::Config::JailCategoryDescrptionScript $category 2>$TmpFile";
        print "makecommand = $makecommand\n";

Let’s try that from the command line:

[16:57 dvl-ingress01 dvl ~/modules] % /usr/local/bin/sudo /usr/sbin/jexec freshports /usr/ports make-category-comment.sh fileservers
jexec: execvp: /usr/ports: Permission denied

No, that’s wrong.

[16:59 dvl-ingress01 dvl ~/modules] % /usr/local/bin/sudo /usr/sbin/jexec freshports /make-category-comment.sh /usr/ports fileservers
cd: /usr/ports/fileservers: No such file or directory
make: cannot open /usr/ports/fileservers/Makefile.

make: stopped in /

I’ve been doing that a lot, typing fileservers when it needs to be filesystems.

[16:59 dvl-ingress01 dvl ~/modules] % /usr/local/bin/sudo /usr/sbin/jexec freshports /make-category-comment.sh /usr/ports filesystems
File systems and related utilities

That’s exactly what we need to see.

That means I need to make this change to the code, swapping two parameters.

-	my $makecommand = "/usr/local/bin/sudo /usr/sbin/jexec $FreshPorts::Config::JailName $REPODIR_CHROOT $FreshPorts::Config::JailCategoryDescrptionScript $category 2>$TmpFile";
+	my $makecommand = "/usr/local/bin/sudo /usr/sbin/jexec $FreshPorts::Config::JailName $FreshPorts::Config::JailCategoryDescrptionScript $REPODIR_CHROOT $category 2>$TmpFile";

Next step: rerun the commit

Let’s process the commit we had running (look above for load_xml_into_db.pl) and which I terminate here:

[17:07 dvl-ingress01 dvl ~/modules] % sudo kill -KILL 47970

[17:10 dvl-ingress01 dvl ~freshports/message-queues/spooling] % echo /usr/local/bin/perl /usr/local/libexec/freshports/load_xml_into_db.pl /var/db/ingress/message-queues/incoming/2024.11.07.10.49.25.000000.43862c0fcfc10374266bf69552e002212a3061a1.xml | sudo su -fm freshports
grabbing categories from disk
'accessibility arabic archivers astro audio benchmarks biology cad chinese comms converters databases deskutils devel distfiles dns editors emulators filesystems finance french ftp games german graphics hebrew hungarian irc japanese java korean lang mail math misc multimedia net net-im net-mgmt net-p2p news polish ports-mgmt portuguese print russian science security shells sysutils textproc ukrainian vietnamese www x11 x11-clocks x11-drivers x11-fm x11-fonts x11-servers x11-themes x11-toolkits x11-wm'

....

about to git checkout = '/usr/local/bin/sudo /usr/local/libexec/freshports/git-checkout.sh /jails/freshports/usr/ports 43862c0fcfc10374266bf69552e002212a3061a1 2>&1'
git checkout result = 256
that git checkout failed.  What do to?


config.sh not found by test-master-port.sh...

So why is the checkout failing? Let’s try that manually.

[17:10 dvl-ingress01 dvl ~freshports/message-queues/spooling] % /usr/local/bin/sudo /usr/local/libexec/freshports/git-checkout.sh /jails/freshports/usr/ports 43862c0fcfc10374266bf69552e002212a3061a
config.sh not found by test-master-port.sh...

Yeah, that’s not right. Why? And why test-master-port.sh?

Seems that was a copy/paste error, now fixed with:

[17:23 dvl-ingress01 dvl ~/scripts] % svn di git-checkout.sh
Index: git-checkout.sh
===================================================================
--- git-checkout.sh	(revision 6088)
+++ git-checkout.sh	(working copy)
@@ -8,7 +8,7 @@
 
 if [ ! -f config.sh ]
 then
-	echo "config.sh not found by test-master-port.sh..."
+	echo "config.sh not found by $0..."
 	exit 1
 fi
 
[17:23 dvl-ingress01 dvl ~/scripts] %

Now let’s try that again, this time, as the freshports user.

$ /usr/local/bin/sudo /usr/local/libexec/freshports/git-checkout.sh /jails/freshports/usr/ports 43862c0fcfc10374266bf69552e002212a3061a
config.sh not found by /usr/local/libexec/freshports/git-checkout.sh...

Oh yes, we have to be in the right directory.

$ cd /usr/local/libexec/freshports
$ ls -l config.sh
lrwxr-xr-x  1 root dvl 35 Feb 11  2024 config.sh -> /usr/local/etc/freshports/config.sh
$ /usr/local/bin/sudo /usr/local/libexec/freshports/git-checkout.sh /jails/freshports/usr/ports 43862c0fcfc10374266bf69552e002212a3061a
num of params = 2
/usr/local/bin/git fetch
remote: Enumerating objects: 292, done.
remote: Counting objects: 100% (292/292), done.
remote: Compressing objects: 100% (189/189), done.
remote: Total 192 (delta 102), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (192/192), 41.52 KiB | 988.00 KiB/s, done.
Resolving deltas: 100% (102/102), completed with 59 local objects.
From https://git.FreeBSD.org/ports
   e8a12c12540c..2001e03f9c06  main       -> origin/main
/usr/local/bin/git checkout 43862c0fcfc10374266bf69552e002212a3061a
Previous HEAD position was e8a12c12540c sysutils/fusefs-httpdirfs: Return to pool
HEAD is now at 43862c0fcfc1 filesystems/zrepl*: unbreak fetch after move to filesystems

OK, that worked. Good. It’s directory based.

Not shown, but the next run mentioned that the commit had already been processed, so I deleted it so we could run it again:

[12:13 pro04 dvl ~] % pg03
Last login: Thu Nov  7 13:07:17 2024 from pro03.startpoint.vpn.unixathome.org
[17:13 pg03 dvl ~] % psql freshports.dvl
psql (16.4)
Type "help" for help.

freshports.dvl=# begin; delete from commit_log where message_id = '43862c0fcfc10374266bf69552e002212a3061a1';
BEGIN
DELETE 1
freshports.dvl=*# commit;
COMMIT
freshports.dvl=#

Now, this run went well:

$ /usr/local/bin/perl /usr/local/libexec/freshports/load_xml_into_db.pl /var/db/ingress/message-queues/incoming/2024.11.07.10.49.25.000000.43862c0fcfc10374266bf69552e002212a3061a1.xml
grabbing categories from disk
'accessibility arabic archivers astro audio benchmarks biology cad chinese comms converters databases deskutils devel distfiles dns editors emulators filesystems finance french ftp games german graphics hebrew hungarian irc japanese java korean lang mail math misc multimedia net net-im net-mgmt net-p2p news polish ports-mgmt portuguese print russian science security shells sysutils textproc ukrainian vietnamese www x11 x11-clocks x11-drivers x11-fm x11-fonts x11-servers x11-themes x11-toolkits x11-wm'
categories already loaded
about to process
finished setting up the Parser
Processing file [/var/db/ingress/message-queues/incoming/2024.11.07.10.49.25.000000.43862c0fcfc10374266bf69552e002212a3061a1.xml]...
parsing file now


 *** start of all updates ***


 *** end of all updates *** 
source in getSource is 'git'
That commit is of Type: 'git'
EOF

...

adding that commit date to the daily summary refresh list
sql = select DailySummaryDateAdd('2024-11-07');
No errors found during that commit
Observer has noticed that the update for 43862c0fcfc10374266bf69552e002212a3061a1 has finished.
Observer will clear the following ports from cache after the commit:
filesystems/zrepl
filesystems/zrepl-dsh2dsh
Observer will clear the following files from cache after the commit:
we are ignoring /ports/head/filesystems/zrepl-dsh2dsh/distinfo for cache clearing
we are ignoring /ports/head/filesystems/zrepl/distinfo for cache clearing
*** end of items to be cleared


 *** end of all updates *** 
Observer has noticed that processing has finished.
 now is the commit:
Observer has noticed that a transaction has been committed.
This commit had no files that need to be removed from the cache
EOF

Creating the category when we first see it

I’ll come back to that part of the code after we get the commits running again.

Next steps:

  1. commit the code
  2. promote the code through dev, test, stage, at
Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

Related Posts

Leave a Comment

Scroll to Top