Dan Langille

I've been playing with Open Source software, starting with FreeBSD, since New Zealand Post installed DSL on my street in 1998. From there, I started writing at The FreeBSD Diary, moving my work here after I discovered WordPress. Along the way, I started the BSDCan and PGCon conferences. I slowly moved from software development into full time systems administration and now work for very-well known company who has been a big force in the security industry.

Manual initiation of git commit processing

 Development, git  No Responses »
Jul 042020
 

This is another post in the FreshPorts-git series.

Yesterday was good progress. This morning (at Sat Jul 4 13:05:10 UTC 2020) I manually initiated another import of commits from git.

It looked like this:

[dan@devgit-ingress01:~/scripts] $ echo /usr/local/libexec/freshports/git-delta.sh | sudo su -fm freshports
/var/db/freshports/ports-jail/var/db/repos/PORTS-head-git
remote: Enumerating objects: 261, done.
remote: Counting objects: 100% (261/261), done.
remote: Compressing objects: 100% (131/131)s done.
remote: Total 265 (delta 149), reused 240 (delta 128), pack-reused 4
Receiving objects: 100% (265/265), 446.50 KiB | 7.09 MiB/s, done.
Resolving deltas: 100% (149/149), completed with 73 local objects.
From https://github.com/freebsd/freebsd-ports
   a35902288517..ecb98d7cd285  branches/2020Q3 -> origin/branches/2020Q3
   25564d38f64f..322037573d9e  master          -> origin/master
   f86664ab0be0..bbe14661ebcb  svn_head        -> origin/svn_head
HEAD is now at f2bfe60090b8 net-mgmt/unifi5: Update to 5.11.46
Updating files: 100% (48503/48503), done.
Previous HEAD position was f2bfe60090b8 net-mgmt/unifi5: Update to 5.11.46
Switched to branch 'master'
Your branch is behind 'origin/master' by 37 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Successfully rebased and updated refs/heads/master.
starting point is d8c85e202575
starting
{'path': PosixPath('/var/db/freshports/ports-jail/var/db/repos/PORTS-head-git'), 'commit': 'd8c85e202575', 'output': PosixPath('/var/db/freshports/message-queues/incoming'), 'repo': 'ports', 'os': 'FreeBSD', 'force': False, 'log_level': 'INFO'}
using repo at  /var/db/freshports/ports-jail/var/db/repos/PORTS-head-git
322037573d9ecbf0f165556df2e349caeaef0534 ba205b385c9b0e91f1f1080655f1ba0c2e1fe7a8 68435f7018ede781c6ba35a7141f1544534e05c4 0d7872f4f69ec9e7b5f7e3298350295301bce70c f5872271c8b9d14e29a6932defcc018d1723194e 918b387b2ed21dd299c4bc966390f3487c7e34bb 6c6db5424c903041d26ea2d2e229af2b1f1d70d6 3061da4ba638f5f9a66818f1b5dec41877f8582b 56a0228b872c9481e823352ac055a0279c362798 55cc857149775dd6ba0b7b08c40b1e6819b9d202 3946cedbad37f7be74dad0648c8935bac1b5465e 882d8de6e0199c6665f3cbe80713fceab235ac92 15f27776ba4f1265590daac8233ff5b9e3aa2197 8475003fd4dbffee1321a5d6e20a0aa750ea5159 b85fb997401b3dd3b71f4b77613adb53a163091d 753eac4badbe2c0a20c6e4c76a228278e7aa30f7 7669079c8ba321c34459b0e1d77c5fccdb4b94e0 43a8a92f64260db919d59824a83873c2fc6be57a e8cdd0bec45d1401594c2a401745e966cde83dcc e250ee3aa03e3bff627d437d68d542adda7ac686 7af772923aea0850ca4fd1fd0851f33586f2b751 0616869d4329ed675dfa0389c7cbb6ee20bcbf31 d3b846b034254602ac214d969c45b62c7dc0fae5 b4a577fe3ccf1188accc53ad9e6a265f80d21b83 926903c161c402218390429f587f9175ec12b31d ecb0127a683f6952ad6ae3b77baf9a310e8499f8 cec47cde28d46071d527ab4d761fc6262cf90cad 0d83d083c00de50dcfb56d61b1095a4aa1d008dc 2f2e89bcc90d64866861a871e546939ab9a8b4d8 c3f75bbcea90b5aca473c5db897f734dc91e8ffb 8c2aef4fd7eab6577fd744749daa88b24edf45d4 8b0acc2a0e75f1e774b940b2966bcab3bce877d9 16de7316ca6bfc7eb2f5f2515408dcf6ad1145f6 b483cab42afe32c814d51cd898fdfd54f8ccf10d cb4e25ed898c489b7c37c929438354f25523d673 da4a6913b642bb4e490d2f91d9068ed4aac97e8f
starting for loop using  d8c85e202575
finished for loop
Would have ran git-to-freshport.py starting at: d8c85e202575
[dan@devgit-ingress01:~/scripts] $

This created one XML file for each git commit. The files are dropped into the ~freshports/messages-queues/incoming directory. The freshports-git daemon (well, it’s a long running shell script, managed via daemontools) notices the new files and processed them into the FreshPorts database.

Compare the commits under these two version of the same port:

devgit, for now, is processing both svn and git commits. This is for development and comparison purposes only.

Being able to process both/either streams is good. I don’t know yet how to avoid duplicate commits once we change over.

Setting up a FreeBSD ports git repo for FreshPorts usage

 Development, git  No Responses »
Jun 302020
 

I’m documenting this as a go along because I’m sure I’ll be doing something that I later wish I had written down.

Create the repo

This creates a FreeBSD git-based port repo.

echo git clone https://github.com/freebsd/freebsd-ports.git /var/db/freshports/ports-jail/var/db/repos/PORTS-head-git | sudo su -l freshports

The latest local commit

This tells us the hash of the latest commit contained within this local copy of the repo.

Let’s call this the base commit.

$ cd PORTS-head-git
[dan@devgit-ingress01:/var/db/freshports/ports-jail/var/db/repos/PORTS-head-git] $ git log -n 1
commit 240cab957e5c13c50ed539ed7bc7d511bb1c3368 (HEAD -> master, origin/master, origin/HEAD)
Author: yuri 
Date:   Tue Jun 30 16:10:10 2020 +0000

    math/fplll: Update 5.3.2 -> 5.3.3
    
    Reported by:    portscout

Now, we do this as the freshports user.

Pull the latest details:

[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ git pull
warning: Pulling without specifying how to reconcile divergent branches is
discouraged. You can squelch this message by running one of the following
commands sometime before your next pull:

  git config pull.rebase false  # merge (the default strategy)
  git config pull.rebase true   # rebase
  git config pull.ff only       # fast-forward only

You can replace "git config" with "git config --global" to set a default
preference for all repositories. You can also pass --rebase, --no-rebase,
or --ff-only on the command line to override the configured default per
invocation.

Already up to date.
[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$

Let’s try that recommended command:

[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ git config pull.rebase false
[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ git pull
Already up to date.

Wait a bit longer for more commits

Some time later:

[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ git pull
remote: Enumerating objects: 345, done.
remote: Counting objects: 100% (308/308), done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 150 (delta 94), reused 137 (delta 81), pack-reused 0
Receiving objects: 100% (150/150), 30.17 KiB | 9.00 KiB/s, done.
Resolving deltas: 100% (94/94), completed with 70 local objects.
From https://github.com/freebsd/freebsd-ports
   240cab957e5c..008e30ab98fb  master          -> origin/master
   6c5a4e63b897..f412caa04aa1  branches/2020Q2 -> origin/branches/2020Q2
   3d3db8513a0a..a70ca3962b88  svn_head        -> origin/svn_head
Updating 240cab957e5c..008e30ab98fb
Fast-forward
 audio/jamulus/Makefile                                        |    2 +-
 audio/jamulus/distinfo                                        |    6 +-
 biology/bamtools/Makefile                                     |    2 +-
 databases/py-redis/Makefile                                   |    2 +-
 databases/py-redis/distinfo                                   |    6 +-
 devel/cargo-c/Makefile                                        |   68 +--
 devel/cargo-c/distinfo                                        |  138 +++---
 devel/cmake-gui/Makefile                                      |    1 +
 devel/cmake/Makefile                                          |    1 +
 devel/opendht/Makefile                                        |    2 +-
 graphics/open3d/Makefile                                      |    2 +-
 graphics/ossim/Makefile                                       |    2 +-
 lang/solidity/Makefile                                        |    1 +
 lang/swi-pl/Makefile                                          |   76 +--
 lang/swi-pl/distinfo                                          |    6 +-
 lang/swi-pl/files/extra-patch-skip-gnu-qsort                  |   55 ---
 lang/swi-pl/files/patch-packages_clib_cmake_FindLibUUID.cmake |   14 +
 lang/swi-pl/pkg-plist                                         | 3677 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------------------------------------------------
 math/cadabra2/Makefile                                        |    1 +
 math/libqalculate/Makefile                                    |    6 +-
 math/libqalculate/distinfo                                    |    8 +-
 math/libqalculate/pkg-plist                                   |    2 +-
 math/py-pynndescent/Makefile                                  |    2 +-
 math/py-pynndescent/distinfo                                  |    6 +-
 math/py-uncertainties/Makefile                                |    2 +-
 math/py-uncertainties/distinfo                                |    6 +-
 math/qalculate/Makefile                                       |    2 +-
 math/qalculate/distinfo                                       |    6 +-
 math/qalculate/pkg-plist                                      |   15 +-
 misc/ezc3d/Makefile                                           |    2 +-
 misc/ezc3d/distinfo                                           |    6 +-
 misc/ignition-fuel-tools/Makefile                             |    2 +-
 multimedia/libopenshot/Makefile                               |    1 +
 net-p2p/eiskaltdcpp-daemon/Makefile                           |    2 +-
 net/coturn/Makefile                                           |    2 +-
 net/coturn/distinfo                                           |    6 +-
 net/flower/Makefile                                           |    2 +-
 net/libjson-rpc-cpp/Makefile                                  |    2 +-
 science/py-tensorflow/Makefile                                |    2 +-
 sysutils/fusefs-securefs/Makefile                             |    1 +
 www/chromium/Makefile                                         |    1 +
 www/domoticz/Makefile                                         |    1 +
 www/iridium/Makefile                                          |    2 +-
 www/qt5-webengine/Makefile                                    |    1 +
 x11-themes/qtcurve-kf5/Makefile                               |    2 +-
 x11-themes/qtcurve-qt5/Makefile                               |    2 +-
 x11-themes/qtcurve/Makefile                                   |   23 +-
 x11-themes/qtcurve/distinfo                                   |    8 +-
 x11-themes/qtcurve/pkg-plist.kde4                             |   21 -
 x11-themes/qtcurve/pkg-plist.utils                            |   27 ++
 x11/polybar/Makefile                                          |    1 +
 x11/waybar/Makefile                                           |    2 +-
 52 files changed, 2148 insertions(+), 2088 deletions(-)
 delete mode 100644 lang/swi-pl/files/extra-patch-skip-gnu-qsort
 create mode 100644 lang/swi-pl/files/patch-packages_clib_cmake_FindLibUUID.cmake
 delete mode 100644 x11-themes/qtcurve/pkg-plist.kde4
[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$

OK, now we have more commits!

List all the commits since the base commit

After a git clone https://github.com/FreshPorts/git_proc_commit I modified git_proc_commit/git-show-commit.sh:

[dan@devgit-ingress01:~/src/git_proc_commit] $ git diff
diff --git a/git-show-commit.sh b/git-show-commit.sh
index fb20d30..4385933 100755
--- a/git-show-commit.sh
+++ b/git-show-commit.sh
@@ -1,14 +1,14 @@
 #!/bin/sh
 
 # the latest commit we have processed
-latest="3bd153c2494182bb89915e6fc9222288c154285f" # [NEW PORT]: devel/py-oci - Wed Sep 18 17:14:34 2019 +0000
+latest="240cab957e5c13c50ed539ed7bc7d511bb1c3368" # [NEW PORT]: devel/py-oci - Wed Sep 18 17:14:34 2019 +0000
 
 # before running this script:
 # mkdir ~/src
 # cd ~/src
 # git clone git@github.com:freebsd/freebsd-ports.git
 
-repo_dir=~/src/freebsd-ports
+repo_dir=~freshports/ports-jail/var/db/repos/PORTS-head-git
 
 GIT="/usr/local/bin/git"
 
[dan@devgit-ingress01:~/src/git_proc_commit] $

Basically, the path and the commit hash were updated. That hash I added in was obtained via the git log -n 1 command.

When running it:

[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ ~dan/src/git_proc_commit/git-show-commit.sh
008e30ab98fba636554ada737157baa84debedb3
46560d7e66714e5d574c69e42ab408f3181c1614
1c7120178a876f6daa217abd64cf8b715de46bd8
12ad02023e15ffeaeca60c78c64c11f9736bbac0
f6a1f3e3c8e17a09dbffefd8debcf3c44861e713
99d2fba63577a91850c9b311d0028c461a4a0b29
42855a0b23b363889207c7d0de5f122982bce37d
21e3f0999b283d1a440c7d0d078b325b138874f2
4046ad10a608fdcb2d6bd71a8e5d4361ad0dc069
4983ea53d2c3b0dfc377c87d6529ca18c76ce229
6f21f15631a6b851cc2f1fc8b2e59c64d808e94f
164aa7374784c77ea68d79875ab3c79950802f4b

Those are the commits which occurred since the previous git pull.

With this information, we can iterate over the commits, processing them one at a time.

First script

This was used for cvs commits, and then svn commits. Now it’s for git commits. This is untested. I’m out of time.

[dan@devgit-ingress01:~/scripts] $ svn di
Index: freebsd-git.sh
===================================================================
--- freebsd-git.sh	(revision 5375)
+++ freebsd-git.sh	(working copy)
@@ -1,25 +1,25 @@
 #!/bin/sh
 #
-# $Id: freebsd-cvs.sh,v 1.9 2011-08-15 16:31:56 dan Exp $
+# $Id: freebsd-git.sh,v 1.9 2011-08-15 16:31:56 dan Exp $
 #
 # Copyright (c) 1999-2003 DVL Software
 #
-# Process a raw mail message by converting it to XML, then importing it into
+# Process a raw git log  message by converting it to XML, then importing it into
 # the database.
 #
-# Takes a file name as a parameter
+# Takes a git commit hash a parameter
 #
-LOGGERTAG="freebsd-cvs.sh"
+LOGGERTAG="freebsd-git.sh"
 
 if [ $# -ne 1 ]
 then
-   echo $0 : usage $0 FILE
+   echo $0 : usage $0 COMMIT_HASH
    exit 1
 fi
 
 if [ ! -f config.sh ]
 then
-	echo "config.sh not found by freebsd-cvs.sh..."
+	echo "config.sh not found by freebsd-git.sh..."
 	exit 1
 fi
 
@@ -37,18 +37,23 @@
 
 ${LOGGER} -t ${LOGGERTAG} "$0 invoked, using XML='${XML}' and OUTPUT='${OUTPUT}'"
 
-PATHNAME=$1
+COMMIT_HASH=$1
 
-FILE=`basename ${PATHNAME}` 
+PROCESS_ID=${$}
+FILE=`date +%Y.%m.%d.%H.%M.%S`.$PROCESS_ID.${COMMIT_HASH}.txt
 
 #
 # convert the raw file to XML
 #
-${LOGGER} -t ${LOGGERTAG} "$0 converting to XML via process_mail.pl"
-${LOGGER} -t ${LOGGERTAG} /usr/local/bin/perl ${SCRIPTDIR}/process_mail.pl from ${PATHNAME} into ${XML}/${FILE}.xml errors to ${XML}/${FILE}.errors
+${LOGGER} -t ${LOGGERTAG} "$0 converting to XML via git-to-freshports-xml.py"
 
-/usr/local/bin/perl ${SCRIPTDIR}/process_mail.pl < ${PATHNAME} >    \
-       ${XML}/${FILE}.xml 2>${XML}/${FILE}.errors
+echo 
+# output file 1to commit_hash.process_id so that if we process the same hash again, it does not conflict
+${LOGGER} -t ${LOGGERTAG} ${SCRIPTDIR}/git-to-freshports-xml.py --path ${FRESHPORTS_JAIL_BASE_DIR}${PORTSDIR} --commit ${COMMIT_HASH} --output ${XML}/${FILE}.xml
+
+exit
+
+${SCRIPTDIR}/git-to-freshports-xml.py --path ${FRESHPORTS_JAIL_BASE_DIR}${PORTSDIR} --commit ${COMMIT_HASH} --output ${XML}/${FILE}.xml 2>${XML}/${FILE}.errors
 RESULT=$?
 
 if [ -f ${XML}/${FILE}.errors ]
@@ -68,8 +73,7 @@
 
 ${LOGGER} -t ${LOGGERTAG} "$0 loading that XML into the database via load_xml_into_db.pl"
 
-/usr/local/bin/perl ${SCRIPTDIR}/load_xml_into_db.pl ${XML}/${FILE}.xml > \
-               ${OUTPUT}/${FILE}.loading 2>${OUTPUT}/$FILE.errors
+/usr/local/bin/perl ${SCRIPTDIR}/load_xml_into_db.pl ${XML}/${FILE}.xml ${OUTPUT}/${FILE}.loading 2>${OUTPUT}/$FILE.errors
 RESULT=$?
 
 if [ -f ${OUTPUT}/$FILE.errors ]

What’s next

I need to alter the script which invokes freebsd-git.sh so that it passes in the git commit hash.

Something like this, with $i set to something obtained from git-show-commit.sh as shown above.

[dan@devgit-ingress01:~/src/fp-daemon] $ svn di
Index: fp-daemon.sh
===================================================================
--- fp-daemon.sh	(revision 5375)
+++ fp-daemon.sh	(working copy)
@@ -92,7 +92,7 @@
 				else
 					${LOGGER} -t ${LOGGERTAG} "processing ${i}"
 
-					/bin/sh ./freebsd-cvs.sh ${i}
+					/bin/sh ./freebsd-git.sh ${i}
 
 					RESULT=$?
 					${LOGGER} -t ${LOGGERTAG} "result=$RESULT"
[dan@devgit-ingress01:~/src/fp-daemon] $

Small steps.

ZFS when you’re not doing ZFS

 packages  No Responses »
Jun 282020
 

Recently, a new use of ZFS snapshots was introduced for a cache on FreshPorts. This approach involves creating a snapshot on the empty cache, then rolling back to that snapshot when the cache needs to be cleared.

This idea came from this Tweet after much discussion on how to properly and safely delete directory contents without hitting a race condition. The zfs rollback is much neater.

This approach is now in place on all FreshPorts servers.

Except one

The original FreshPorts server was in a small tower chassis. The first deployed-to-a-datacenter server was a dual-Opteron with 8 x 74GB Raptor drives. This host was deployed in 2006 and is still in use. It remained the primary FreshPorts server until November 2017 when it was replaced by a new system from iXsystems.

More about the Opteron server.

This server is not running ZFS and hardware RAID. Converting to ZFS remotely is not going to happen.

I have been putting off upgrading this server to the latest FreshPorts code because of this.

Last night I realized I could deploy that code by creating a ZFS file system within a regular UFS file.

The ZFS file

From man zpool, don’t do this at home:

     file    A regular file. The use of	files as a backing store is strongly
             discouraged. It is	designed primarily for experimental purposes,
             as	the fault tolerance of a file is only as good the file system
             of	which it is a part. A file must	be specified by	a full path.

For this specific purpose, I am accepting the risk.

First step, enable and start ZFS.

[dan@supernews:~] $ sudo sysrc zfs_enable="YES"
[dan@supernews:~] $ sudo service zfs start
[dan@supernews:~] $ zfs list
no datasets available

When I went looking for how to create the file, I found Ubuntu 16.04 – Using Files To Test ZFS and the truncate command was used.

How big should this file be? Let’s look at production.

[dan@x8dtu-nginx01:~] $ zfs list main_tank/data/freshports/backend/cache/packages
NAME                                               USED  AVAIL  REFER  MOUNTPOINT
main_tank/data/freshports/backend/cache/packages  7.90M  3.47T  7.84M  /var/db/freshports/cache/packages

[dan@x8dtu-nginx01:~] $ sudo du -ch -d0  /var/db/freshports/cache/packages
8.3M	/var/db/freshports/cache/packages
8.3M	total

Let’s say that is 9MB.

How many files are in there?

[dan@x8dtu-nginx01:~] $ sudo find /var/db/freshports/cache/packages | wc -l
    1356

There are packages for 1356 ports cached. Let’s say there are 40,000 packages. That would be 9MB / 1356 * 40,000 = 265 MB

Let’s call it 1GB for good measure. This is only for caching.

After consulting man truncate, I settled on this command:

[dan@supernews:~/tmp] $ truncate -s 1G packages
[dan@supernews:~/tmp] $ ls -l packages
-rw-r--r--  1 dan  dan  1073741824 Jun 28 12:59 packages
[dan@supernews:~/tmp] $

Before I start messing with the website cache, let’s turn off the services which use it:

[dan@supernews:/var/db/freshports/cache] $ sudo service nginx stop
Stopping nginx.
Waiting for PIDS: 78270.

[dan@supernews:/var/db/freshports/cache] $ sudo svc -d /var/service/freshports
[dan@supernews:/var/db/freshports/cache] $ sudo svc -d /var/service/fp-listen
[dan@supernews:/var/db/freshports/cache] $

The two svc items are the commit processing daemon and the cache clearing daemon, respectively.

This moves the file into place:

[dan@supernews:/var/db/freshports] $ sudo mv ~/tmp/packages freshports-zfs
[dan@supernews:/var/db/freshports] $ sudo chown root:wheel freshports-zfs

Creating the zpool

This creates the zpool:

$ sudo zpool create freshports /var/db/freshports/freshports-zfs
$ zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freshports   960M  87.5K   960M        -         -     0%     0%  1.00x  ONLINE  -

It works, just like any other zpool:

[dan@supernews:/var/db/freshports/cache] $ zpool status
  pool: freshports
 state: ONLINE
  scan: none requested
config:

	NAME                                 STATE     READ WRITE CKSUM
	freshports                           ONLINE       0     0     0
	  /var/db/freshports/freshports-zfs  ONLINE       0     0     0

errors: No known data errors
[dan@supernews:/var/db/freshports/cache] $ sudo zpool scrub freshports
[dan@supernews:/var/db/freshports/cache] $ zpool status
  pool: freshports
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:00 with 0 errors on Sun Jun 28 14:29:26 2020
config:

	NAME                                 STATE     READ WRITE CKSUM
	freshports                           ONLINE       0     0     0
	  /var/db/freshports/freshports-zfs  ONLINE       0     0     0

errors: No known data errors
[dan@supernews:/var/db/freshports/cache] $

Next up, this zpool needs a filesystem.

Creating the filesystem

I don’t need atime, we don’t need compression here, and let’s set the mountpoint.

Compression is usually a good thing. I’m deliberately choosing not to use it. I can’t justify. It just feels wrong to do this on top of UFS. This isn’t anything about UFS. It’s just me.

$ sudo zfs create -o atime=off -o compression=off -o mountpoint=/var/db/freshports/cache/packages freshports/packages

$ zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
freshports            175K   832M    23K  /freshports
freshports/packages    23K   832M    23K  /var/db/freshports/cache/packages

Oh wait, I don’t like having /freshports mounted.

$ sudo zfs set mountpoint=none canmount=off freshports

$ zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
freshports            188K   832M    23K  none
freshports/packages    23K   832M    23K  /var/db/freshports/cache/packages

What do the permissions look like?

$ ls -l /var/db/freshports/cache
total 27619
drwxrwxr--  27 www         freshports       512 Apr 20 08:19 categories
drwxrwxr-x   3 www         freshports       512 Jan 23 14:22 daily
drwxr-xr-x   2 www         freshports       512 Apr 20 08:21 general
drwxr-xr-x   2 freshports  freshports       512 Jun 28 14:33 html
drwxrwxr-x   2 www         freshports       512 Jun 28 06:10 news
drwxr-xr-x   2 root        wheel              2 Jun 28 14:34 packages
drwxrwxr--  14 www         freshports       512 Jun 23 12:00 packages.old
drwxr-xr-x   2 www         freshports       512 Nov  9  2018 pages
drwxrwxr-x  41 www         freshports      2048 May 18 22:42 ports
-rw-r--r--   1 www         www         28234059 Jun  6 05:42 searchlog.txt
drwxrwxr-x   2 www         freshports       512 Jun 28 14:33 spooling
[dan@supernews:/var/db/freshports/cache] $

That’s wrong, fixing it:

$ sudo chown www:freshports packages

Snapshot now!

I must snapshot now, before making any data changes which I do not want to rollback.

$ sudo zfs snapshot freshports/packages@empty

Edit 2020-08-25

Missing from this post, as I discovered last night, delegation.

Right now, it does not work:

$ echo zfs rollback freshports/packages@empty | sudo su -fm freshports
cannot rollback 'freshports/packages': permission denied

I must allow the freshports user to do the rollback.

sudo zfs allow freshports rollback freshports/packages

Now it does:

$ echo zfs rollback freshports/packages@empty | sudo su -fm freshports
$

It only took me two months and a failed FreshPorts server to find out.

Moving over the data

Now I move over the existing cache:

$ sudo mv -i packages.old/ packages/
$ cd packages/
$ ls
packages.old

Or… not. Damn.

$ sudo mv packages.old/* 
$ sudo rmdir packages.old/
$ ls
comms        devel        java         mail         multimedia   www
databases    emulators    lang         misc         net-p2p      x11-toolkits

Back into service

Let’s start up everything I stopped.

$ sudo svc -u /var/service/freshports
$ sudo svc -u /var/service/fp-listen
$ sudo service nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.

Next?

Next, this server needs to have the latest code installed. That’s not for today.

git – the next FreshPorts project

 Development, git  No Responses »
Jun 212020
 

git is coming to FreeBSD.

The initial FreshPorts work for git started in September, then was abandoned for unknown reasons. It just fell off the RADAR. This week, it came back to my attention.

References

Some of the initial research was over Twitter. I’m saving those references here for future reference.

Today, I created new new FreeBSD jails for this work:

  1. devgit-ingress01 – for processing of incoming FreeBSD commits
  2. devgit-nginx01 – for displaying those commits

As the names suggests, these jails are specific to git. The website is not yet publicly available, but will be soon.

Current status

At present, we have a script which will process “GIT commit entries into XML files digestible by the FreshPorts backend”.

This script has been used to process a git commit.

Current task list

This is the current task list:

  • pull in the local changes from the proof of concept jail – this includes both ingress and www code.
  • Branch the ingress code – the www code has already been branched and is in place on devgit-nginx01
  • get devgit-nginx01 online so the public can see it – see https://devgit.freshports.org
  • Incorporate the git-to-freshports.py scripts into the FreshPorts automation
  • package the code into git-specific instances
  • Create test, stage, and prod instances for the git-specific code – only dev exists so far
  • Create the database instances for those new environments
  • Adjust the backups to cover the 8 new jails and 4 new databases – freshports.devgit is being backed up
  • create certs for the four new environments

I am sure other tasks will come to mind over the next few days, but I always like to have a starting point.

Welcome to the next adventure.

How often is a new repo available?

 packages  No Responses »
Jun 032020
 

How long does it take for a new repo to come online.

FreshPorts does not have a lot of stats yet, but here is what we have today:

freshports.org=# SELECT abi.name, package_set, inserts, updates, deletes, repo_date
  FROM package_imports PI, abi
 WHERE PI.abi_id = abi.id and repo_date > '2020-05-31' 
 ORDER BY  abi.name, package_set, repo_date;
        name        | package_set | inserts | updates | deletes |       repo_date        
--------------------+-------------+---------+---------+---------+------------------------
 FreeBSD:11:aarch64 | quarterly   |      13 |      80 |       1 | 2020-06-01 20:41:00+00
 FreeBSD:11:amd64   | latest      |      12 |     222 |      28 | 2020-06-01 00:43:00+00
 FreeBSD:11:amd64   | quarterly   |       0 |      10 |       0 | 2020-05-31 22:42:00+00
 FreeBSD:11:amd64   | quarterly   |       0 |      10 |       0 | 2020-06-02 07:32:00+00
 FreeBSD:11:armv6   | quarterly   |       0 |       4 |       0 | 2020-06-02 09:59:00+00
 FreeBSD:11:i386    | latest      |      14 |     210 |      25 | 2020-05-31 19:15:00+00
 FreeBSD:11:i386    | quarterly   |       0 |      10 |       0 | 2020-05-31 16:20:00+00
 FreeBSD:11:i386    | quarterly   |       0 |      10 |       0 | 2020-06-02 07:12:00+00
 FreeBSD:11:mips    | quarterly   |       0 |       1 |       0 | 2020-05-31 02:45:00+00
 FreeBSD:11:mips    | quarterly   |       0 |       2 |       0 | 2020-06-02 02:46:00+00
 FreeBSD:11:mips64  | quarterly   |       0 |       1 |       0 | 2020-05-31 08:32:00+00
 FreeBSD:11:mips64  | quarterly   |       0 |       2 |       0 | 2020-06-02 08:33:00+00
 FreeBSD:12:amd64   | latest      |      13 |     225 |      22 | 2020-05-31 23:49:00+00
 FreeBSD:12:amd64   | quarterly   |       0 |      10 |       0 | 2020-05-31 22:08:00+00
 FreeBSD:12:amd64   | quarterly   |       0 |      10 |       0 | 2020-06-02 07:21:00+00
 FreeBSD:12:armv6   | quarterly   |       1 |       6 |      27 | 2020-05-31 18:45:00+00
 FreeBSD:12:armv6   | quarterly   |      24 |       4 |       0 | 2020-06-02 20:45:00+00
 FreeBSD:12:armv7   | quarterly   |      11 |       7 |      36 | 2020-05-31 01:22:00+00
 FreeBSD:12:armv7   | quarterly   |      31 |       4 |       0 | 2020-06-02 12:13:00+00
 FreeBSD:12:i386    | latest      |      15 |     210 |      25 | 2020-05-31 19:23:00+00
 FreeBSD:12:i386    | quarterly   |       0 |      10 |       1 | 2020-05-31 16:21:00+00
 FreeBSD:12:i386    | quarterly   |       0 |      10 |       0 | 2020-06-02 07:07:00+00
 FreeBSD:12:mips    | quarterly   |       0 |       1 |       0 | 2020-05-31 04:21:00+00
 FreeBSD:12:mips    | quarterly   |       0 |       2 |       0 | 2020-06-02 04:24:00+00
 FreeBSD:12:mips64  | quarterly   |       0 |       1 |       0 | 2020-05-31 15:57:00+00
 FreeBSD:12:mips64  | quarterly   |       0 |       2 |       0 | 2020-06-02 15:58:00+00
 FreeBSD:13:amd64   | latest      |      39 |     347 |      40 | 2020-05-31 04:02:00+00
 FreeBSD:13:amd64   | latest      |      25 |     428 |      59 | 2020-06-03 04:29:00+00
 FreeBSD:13:i386    | latest      |      16 |     210 |      29 | 2020-05-31 02:56:00+00
 FreeBSD:13:i386    | latest      |      31 |     419 |      50 | 2020-06-03 04:30:00+00
 FreeBSD:13:mips64  | latest      |       1 |       5 |       0 | 2020-05-31 09:30:00+00
 FreeBSD:13:mips64  | latest      |       1 |      30 |       5 | 2020-06-02 09:55:00+00
(32 rows)

freshports.org=#

Roughly, it’s every two days.

The value in repo_date is taken from packagesite.txz as found at https://pkg.freebsd.org/FreeBSD:12:amd64/latest/ for example.

What are inserts, updates, deletes?

  1. inserts: The number of packages which were not present in the previous instance of this repo but were in this instances (e.g. new ports).
  2. updates: The number of packages present in the previous instance of this repo but now have different version numbers (e.g. the port has been updated).
  3. deletes: The number of package present in the prevoius instance of this repo, but were not found in the new repo (e.g. port has been deleted).

Graphing this data is a good idea, but we need help getting graphs working again.

Tracking down Wide character in print at /usr/local/lib/perl5/site_perl/FreshPorts/vuxml_parsing.pm line 232

 Bug fixes, vuxml  No Responses »
May 302020
 

I noticed this today:

root       58697  0.0  0.0 10680  2180  -  IJ   12May20  0:00.00 /usr/local/bin/readproctitle service errors: .../site_perl/FreshPorts/vuxml_parsing.pm line 232, <> chunk 1.\nWide character in print at /usr/local/lib/perl5/site_perl/FreshPorts/vuxml_parsing.pm line 232, <> chunk 1.\nWide character in print at /usr/local/lib/perl5/site_perl/FreshPorts/vuxml_parsing.pm line 232, <> chunk 1.\nUndefined subroutine &FreshPorts::CommitterOptIn::RecordErrorDetails called at ./process_vuxml.pl line 124, <> chunk 1.\n

I should monitor that process better.

But first, let’s find the vuxml entry which causes this. We know it’s after 12 May because that’s the date on the process.

Testing

To test the processing of a single vuxml entry, I delete it from the database.

freshports.dev=# begin;
BEGIN
freshports.dev=# delete from vuxml where vid = '0bfcae0b-947f-11ea-92ab-00163e433440';
DELETE 1
freshports.dev=# commit;
COMMIT
freshports.dev=#

To initiate the processing of the vuxml file, I enter this command:

echo touch ~freshports/signals/vuxml ~freshports/signals/job_waiting | sudo su -fm freshports

This will invoke the vuml processing during the normal course of events. If a commit is being processed when the command is entered, the vuxml will not be processed until after that commit.

I could run the commands manually from the command line, but that would not properly exercise the code.

Found it!

I found 59fabdf2-9549-11ea-9448-08002728f74c produces the issue:

[dan@dev-ingress01:/var/service/freshports] $ ps auwwx | grep proc
root       92487  0.0  0.0 10680  2180  -  IJ   13:07    0:00.00 /usr/local/bin/readproctitle service errors: 
........Wide character in print at /usr/local/lib/perl5/site_perl/FreshPorts/vuxml_parsing.pm line 234, <> chunk 1.\n

Reproducing the issue is always a good first goal.

This is line 234:

        print "description:", $self->description(), "\n";

perldoc.perl.org says:

To avoid this warning and to avoid having different output encodings in a single stream, always specify an encoding explicitly, for example with a PerlIO layer:

    binmode STDOUT, ":encoding(UTF-8)";

That code goes into process_vuxml.pl which uses the vuxml_parsing.pm module.

Let’s try that. The steps are:

  1. freshports.dev=# begin; delete from vuxml where vid = '59fabdf2-9549-11ea-9448-08002728f74c'; commit;
BEGIN
DELETE 1
COMMIT
freshports.dev=# 

  2. $ echo touch ~freshports/signals/vuxml ~freshports/signals/job_waiting | sudo su -fm freshports

  3. $ ps auwwx | grep proc
root       34401  0.0  0.0 10680  2180  -  IJ   13:25    0:00.00 /usr/local/bin/readproctitle service errors: 
........................................................................................................
dan        35363  0.0  0.0 11324  2648  4  S+J  13:26    0:00.00 grep proc

No errors!

Packages – now in production

 packages  No Responses »
May 182020
 

On Sunday, the packages project was moved into production. This project was both fun and challenging.

The goal: within 1 hour of the repo being updated, FreshPorts has the packages imported and updated for your viewing pleasure.

Today, I went checking to see if a repo update had occurred. An issue was found and corrected. This post contains the details.

Why post this stuff?

My writing helps me remember and understand what went wrong and helps me avoid similar oversights in the future. It also helps document how the system works and will be useful when someone else takes over FreshPorts.

The repo update

Looking at the backend status page I saw that an update had occurred today at 1800 UTC:

FreeBSD:13:amd64 updated

FreeBSD:13:amd64 updated

Browsing over to the sysutils/bacula9-server page, I could see it was still listing the packages from the 2020-05-16 build:

sysutils/bacula9-server 202-05-16 build

sysutils/bacula9-server 202-05-16 build

Ideas

What follows are my investigations, writtten after the fact of where I looked and what I found.

Was it imported?

First, I checked the logs for events around 1800 UTC. I found plenty of evidence that the repo update had been detected and imported:

May 18 18:00:07 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: Starting up
May 18 18:00:07 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:11:aarch64/latest : 2018-Sep-29 04:24
May 18 18:00:07 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:11:aarch64/quarterly : 2020-May-05 00:02
May 18 18:00:07 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:11:amd64/latest : 2020-May-13 21:25
May 18 18:00:08 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:11:amd64/quarterly : 2020-May-12 05:56
May 18 18:00:08 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:11:i386/latest : 2020-May-13 18:34
May 18 18:00:08 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:11:i386/quarterly : 2020-May-12 05:56
May 18 18:00:08 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:12:aarch64/latest : 2018-Oct-11 05:47
May 18 18:00:08 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:12:aarch64/quarterly : 2020-May-09 17:08
May 18 18:00:08 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:12:amd64/latest : 2020-May-13 20:01
May 18 18:00:08 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:12:amd64/quarterly : 2020-May-12 05:30
May 18 18:00:08 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:12:i386/latest : 2020-May-13 17:58
May 18 18:00:09 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:12:i386/quarterly : 2020-May-12 05:41
May 18 18:00:09 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:13:aarch64/latest : 2020-May-16 06:24
May 18 18:00:09 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:13:aarch64/quarterly : 
May 18 18:00:09 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:13:amd64/latest : 2020-May-18 04:59
May 18 18:00:09 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:13:amd64/quarterly : 
May 18 18:00:09 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:13:i386/latest : 2020-May-18 05:40
May 18 18:00:09 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: checking: FreeBSD:13:i386/quarterly : 
May 18 18:00:09 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: There are 2 new repos ready for import: ['FreeBSD:13:amd64/latest : 2020-May-18 04:59', 'FreeBSD:13:i386/latest : 2020-May-18 05:40']
May 18 18:00:09 x8dtu-ingress01 check_repos_for_new_stuff.py[46975]: finishes
May 18 18:00:10 x8dtu-ingress01 fp-daemon[47094]: yes, there is a job waiting
May 18 18:00:10 x8dtu-ingress01 FreshPorts[47095]: flag '/var/db/freshports/signals/vuxml' not set.  no work for process_vuxml.sh (/usr/local/libexec/freshports) 
May 18 18:00:10 x8dtu-ingress01 FreshPorts[47095]: /var/db/freshports/signals/new_repo_ready_for_import exists.  About to run import_packagesite.py (/usr/local/libexec/freshports) 
May 18 18:00:10 x8dtu-ingress01 import_packagesite.py[47105]: Starting up
May 18 18:00:10 x8dtu-ingress01 import_packagesite.py[47105]: we have 2 repos to process
May 18 18:00:10 x8dtu-ingress01 import_packagesite.py[47105]: importing FreeBSD:13:amd64/latest
May 18 18:00:10 x8dtu-ingress01 import_packagesite.py[47105]: command is: /usr/local/libexec/freshports/fetch-extract-parse-import-one-abi.sh FreeBSD:13:amd64 latest
May 18 18:00:10 x8dtu-ingress01 freshports[47108]: got into /usr/local/libexec/freshports/fetch-extract-parse-import-one-abi.sh
May 18 18:00:10 x8dtu-ingress01 dan[47109]: starting /usr/local/libexec/freshports/fetch-extract-parse-import-one-abi.sh
May 18 18:00:25 x8dtu-ingress01 import-via-copy-packagesite-all-raw-fields.py[47420]: Starting up
May 18 18:00:25 x8dtu-ingress01 syslogd: last message repeated 1 times
May 18 18:00:25 x8dtu-ingress01 import-via-copy-packagesite-all-raw-fields.py[47420]: copying in from packagesite.tsv
May 18 18:00:27 x8dtu-ingress01 import-via-copy-packagesite-all-raw-fields.py[47420]: Finishing
May 18 18:00:27 x8dtu-ingress01 dan[47428]: finished importing FreeBSD:13:amd64/latest
May 18 18:00:27 x8dtu-ingress01 import_packagesite.py[47105]: calling PackagesLastCheckedSetImportDate()
May 18 18:00:27 x8dtu-ingress01 import_packagesite.py[47105]: importing FreeBSD:13:i386/latest
May 18 18:00:27 x8dtu-ingress01 import_packagesite.py[47105]: command is: /usr/local/libexec/freshports/fetch-extract-parse-import-one-abi.sh FreeBSD:13:i386 latest
May 18 18:00:27 x8dtu-ingress01 freshports[47430]: got into /usr/local/libexec/freshports/fetch-extract-parse-import-one-abi.sh
May 18 18:00:27 x8dtu-ingress01 dan[47431]: starting /usr/local/libexec/freshports/fetch-extract-parse-import-one-abi.sh
May 18 18:00:30 x8dtu-ingress01 import-via-copy-packagesite-all-raw-fields.py[47465]: Starting up
May 18 18:00:30 x8dtu-ingress01 syslogd: last message repeated 1 times
May 18 18:00:30 x8dtu-ingress01 import-via-copy-packagesite-all-raw-fields.py[47465]: copying in from packagesite.tsv
May 18 18:00:31 x8dtu-ingress01 import-via-copy-packagesite-all-raw-fields.py[47465]: Finishing
May 18 18:00:31 x8dtu-ingress01 dan[47480]: finished importing FreeBSD:13:i386/latest
May 18 18:00:31 x8dtu-ingress01 import_packagesite.py[47105]: calling PackagesLastCheckedSetImportDate()
May 18 18:00:31 x8dtu-ingress01 import_packagesite.py[47105]: There are 2 repos which need post-import processing: ['FreeBSD:13:amd64:latest', 'FreeBSD:13:i386:latest']
May 18 18:00:31 x8dtu-ingress01 import_packagesite.py[47105]: Finishing
May 18 18:00:31 x8dtu-ingress01 FreshPorts[47095]: Finished running import_packagesite.py (/usr/local/libexec/freshports) 
May 18 18:00:31 x8dtu-ingress01 FreshPorts[47095]: flag '/var/db/freshports/signals/refresh_moved' not set.  no work for process_moved.sh (/usr/local/libexec/freshports) 
May 18 18:00:31 x8dtu-ingress01 FreshPorts[47095]: flag '/var/db/freshports/signals/refresh_updating' not set.  no work for process_updating.sh (/usr/local/libexec/freshports) 
May 18 18:00:31 x8dtu-ingress01 FreshPorts[47095]: /var/db/freshports/signals/new_repo_imported exists.  About to run UpdatePackagesFromRawPackages.py (/usr/local/libexec/freshports) 
May 18 18:00:31 x8dtu-ingress01 UpdatePackagesFromRawPackages.py[47481]: Starting up
May 18 18:00:31 x8dtu-ingress01 UpdatePackagesFromRawPackages.py[47481]: updating packages table for FreeBSD:13:amd64/latest
May 18 18:00:37 x8dtu-ingress01 UpdatePackagesFromRawPackages.py[47481]: updating packages table for FreeBSD:13:i386/latest
May 18 18:02:16 x8dtu-ingress01 UpdatePackagesFromRawPackages.py[47481]: There were 2 repos updated in packages.
May 18 18:02:16 x8dtu-ingress01 UpdatePackagesFromRawPackages.py[47481]: Finishing
May 18 18:02:16 x8dtu-ingress01 FreshPorts[47095]: Finished running UpdatePackagesFromRawPackages.py (/usr/local/libexec/freshports)

Conclusion: data import is working properly.

Was the cache cleared

For cache clearing to occur, the above process issues a PostgreSQL NOTIFY.

My next idea was fp-listen on the x8dtu-nginx01 host was not listening for the new events for clearing the package cache. The logs indicated it was not:

I went looking for the list output upon startup:

May 17 01:38:07 x8dtu-nginx01 fp-listen[42345]: Starting up - this should not occur often
May 17 01:38:07 x8dtu-nginx01 fp-listen[42345]: These are the (event name, script name) pairs we are ready for:
May 17 01:38:07 x8dtu-nginx01 fp-listen[42345]: ('port_updated', 'listen_port')
May 17 01:38:07 x8dtu-nginx01 fp-listen[42345]: ('ports_moved', 'listen_ports_moved')
May 17 01:38:07 x8dtu-nginx01 fp-listen[42345]: ('ports_updating', 'listen_ports_updating')
May 17 01:38:07 x8dtu-nginx01 fp-listen[42345]: ('vuxml', 'listen_vuxml')
May 17 01:38:07 x8dtu-nginx01 fp-listen[42345]: ('category_new', 'listen_category_new')
May 17 01:38:07 x8dtu-nginx01 fp-listen[42345]: ('date_updated', 'listen_date_updated')

Comparing it to dev, I found packages_imported was missing.

May 12 20:36:53 dev-nginx01 fp-listen[59516]: Starting up - this should not occur often
May 12 20:36:53 dev-nginx01 fp-listen[59516]: These are the (event name, script name) pairs we are ready for:
May 12 20:36:53 dev-nginx01 fp-listen[59516]: ('port_updated', 'listen_port')
May 12 20:36:53 dev-nginx01 fp-listen[59516]: ('ports_moved', 'listen_ports_moved')
May 12 20:36:53 dev-nginx01 fp-listen[59516]: ('ports_updating', 'listen_ports_updating')
May 12 20:36:53 dev-nginx01 fp-listen[59516]: ('vuxml', 'listen_vuxml')
May 12 20:36:53 dev-nginx01 fp-listen[59516]: ('category_new', 'listen_category_new')
May 12 20:36:53 dev-nginx01 fp-listen[59516]: ('date_updated', 'listen_date_updated')
May 12 20:36:53 dev-nginx01 fp-listen[59516]: ('packages_imported', 'listen_packages_imported')

Checking the webserver

Checking the webserver, I was about to restart the fp-listen process when I thought to check this output:

$ ps auwwx | grep readproc
root       49860  0.0  0.0  10680  2180  -  IJ   Sun14   0:00.00 /usr/local/bin/readproctitle service errors: ................................................................................................................
..................................................................................cannot open 
'main_tank/data/freshports/backend/cache/packages': missing '@' delimiter in snapshot name\ncannot open 
'main_tank/data/freshports/backend/cache/packages': missing '@' delimiter in snapshot name\n

HAH! So the NOTIFY was received.

The issue was configuration as shown by my change here:

[dan@x8dtu-nginx01:/usr/local/etc/freshports] $ sudo diff -ruN fp-listen.ini~ fp-listen.ini
--- fp-listen.ini~	2020-05-17 13:25:50.000000000 +0000
+++ fp-listen.ini	2020-05-18 19:35:58.964021000 +0000
@@ -18,7 +18,7 @@
 NEWS_CACHE_DIR  = %(BASEDIR)s/cache/news/
 
 # this is the name of the zfs filesystem to rollback, including the snapshot name
-PKG_ZFS_SNAPSHOT    = main_tank/data/freshports/backend/cache/packages
+PKG_ZFS_SNAPSHOT    = main_tank/data/freshports/backend/cache/packages@empty
 
 [flags]
 # used for processing category updates
[dan@x8dtu-nginx01:/usr/local/etc/freshports] $

The snapshot name was missing.

Clearing the cache

I cleared the cache by entering this command:

[dan@x8dtu-nginx01:~] $ echo zfs rollback main_tank/data/freshports/backend/cache/packages@empty | sudo su -fm freshports

Checking the webpage

Now the sysutils/bacula9-server page shows:

sysutils/bacula9-server 18:00

sysutils/bacula9-server 18:00

Now that’s the refreshed data.

What about the missing LISTEN for information?

Remember when I said the host was not listening for the new events? Let’s check farther into the logs:

May 17 12:59:32 x8dtu-nginx01 fp-listen[14300]: Starting up - this should not occur often
May 17 12:59:32 x8dtu-nginx01 fp-listen[14300]: These are the (event name, script name) pairs we are ready for:
May 17 12:59:32 x8dtu-nginx01 fp-listen[14300]: ('port_updated', 'listen_port')
May 17 12:59:32 x8dtu-nginx01 fp-listen[14300]: ('ports_moved', 'listen_ports_moved')
May 17 12:59:32 x8dtu-nginx01 fp-listen[14300]: ('ports_updating', 'listen_ports_updating')
May 17 12:59:32 x8dtu-nginx01 fp-listen[14300]: ('vuxml', 'listen_vuxml')
May 17 12:59:32 x8dtu-nginx01 fp-listen[14300]: ('category_new', 'listen_category_new')
May 17 12:59:32 x8dtu-nginx01 fp-listen[14300]: ('date_updated', 'listen_date_updated')
May 17 12:59:32 x8dtu-nginx01 fp-listen[14300]: ('packages_imported', 'listen_packages_imported')

Yeah, there it is. It was listening. I was looking too early in the logs

A lingering issue

While inspecting the logs, I noticed that fp-listen was restarting often, sometimes hundreds of times a day.

[dan@x8dtu-nginx01:~] $ sudo grep -c 'Starting up - this should not occur often' /var/log/freshports/freshports.log*
/var/log/freshports/freshports.log:0
/var/log/freshports/freshports.log.0:85
/var/log/freshports/freshports.log.1:255
/var/log/freshports/freshports.log.10:0
/var/log/freshports/freshports.log.11:0
/var/log/freshports/freshports.log.12:0
/var/log/freshports/freshports.log.13:0
/var/log/freshports/freshports.log.2:159
/var/log/freshports/freshports.log.3:119
/var/log/freshports/freshports.log.4:236
/var/log/freshports/freshports.log.5:129
/var/log/freshports/freshports.log.6:0
/var/log/freshports/freshports.log.7:0
/var/log/freshports/freshports.log.8:0
/var/log/freshports/freshports.log.9:0

This is an issue I noticed earlier and fixed. I also think it was related to a recent issue.

The problem occurs when trying to clear the cache for for a port. This appears in the logs as:

May 17 12:59:32 x8dtu-nginx01 fp-listen[2856]: removing glob /var/db/freshports/cache/ports/devel/py-setuptools/*

There have been no restarted since May 17 17:01:28. Checking the logs, I see this was intentional, because I restarted the fp-listen process at that time.

May 17 17:01:28 x8dtu-nginx01 sudo[38174]:      dan : TTY=pts/2 ; PWD=/var/db/freshports/cache/packages ; USER=root ; COMMAND=/usr/local/bin/svc -t /var/service/fp-listen

I’m guessing I did that right after adding the required new entry to the database, something like this:

-- insert this so fp-listen knows to listen
freshports.org=# insert into listen_for ( name, script_name ) values ('listen_for', 'ClearPackagesCache');

The original is found in my git repo

Welcome to packages

I’m glad that work is now in production, not just because it was complex, but also because of how seamlessly the data is imported and presented to the user.

If you notice issues, please let me know. If you are a maintainer and you know the repo has a lagged version, please check to see if FreshPorts detects them.

Thank you.

Edit: 2020-05-22

fp-listen was not restarted after the update to fp-listen.ini:

[dan@x8dtu-nginx01:/usr/local/etc/freshports] $ ps auwwx | grep fp-listen
freshports 38176  0.0  0.0  29348  19668  -  SJ   Sun17    0:58.23 /usr/local/bin/python3.7 /usr/local/lib/python3.7/site-packages/fp-listen/fp-listen.pyc
root       49862  0.0  0.0  10696   2192  -  IJ   Sun14    0:00.02 supervise fp-listen
dan        66966  0.0  0.0  11176   2676  0  R+J  20:40    0:00.00 grep fp-listen
[dan@x8dtu-nginx01:/usr/local/etc/freshports] $ ls -l fp-listen.ini
-rw-r-----  1 root  freshports  813 May 18 19:35 fp-listen.ini
[dan@x8dtu-nginx01:/usr/local/etc/freshports] $

The date on the file is after the date on the process. Rookie error.

fp-listen has been restarted.

Edit: 2020-05-22

That’s what we want to see!

May 24 01:00:28 x8dtu-nginx01 fp-listen[67261]: Just woke up! *************
May 24 01:00:28 x8dtu-nginx01 fp-listen[67261]: Got NOTIFY: pid='71564', channel='packages_imported', payload=''
May 24 01:00:28 x8dtu-nginx01 fp-listen[67261]: found key packages_imported
May 24 01:00:28 x8dtu-nginx01 fp-listen[67261]: invoking PackagesCacheClear()
May 24 01:00:28 x8dtu-nginx01 fp-listen[67261]: Time to rollback main_tank/data/freshports/backend/cache/packages@empty
May 24 01:00:30 x8dtu-nginx01 fp-listen[67261]: zfs rollback succeeded.
May 24 01:00:30 x8dtu-nginx01 fp-listen[67261]: Done with PackagesCacheClear()

Done. Closing this issue.

Clearing the packages cache

 Caching, packages  No Responses »
Apr 292020
 

When a new repo is imported, the packages cache needs to be deleted. The cache is on the front-end, the web server. Cache clearing on the web server is handled by the fp-listen daemon.

It does this by using the LISTEN command.

The list of channels it listens to is controlled by the listen_for table:

freshports.test=# \d listen_for
                               Table "public.listen_for"
   Column    |  Type   | Collation | Nullable |                Default                 
-------------+---------+-----------+----------+----------------------------------------
 id          | integer |           | not null | nextval('listen_for_id_seq'::regclass)
 name        | text    |           | not null | 
 script_name | text    |           | not null | 
Indexes:
    "listen_for_pkey" PRIMARY KEY, btree (id)
    "listen_for_name_idx" UNIQUE, btree (name)

freshports.test=#

The values now in place are:

freshports.test=# SELECT name, script_name FROM listen_for ORDER BY name;
      name      |      script_name      
----------------+-----------------------
 category_new   | listen_category_new
 date_updated   | listen_date_updated
 port_updated   | listen_port
 ports_moved    | listen_ports_moved
 ports_updating | listen_ports_updating
 vuxml          | listen_vuxml
(6 rows)

freshports.test=#

I’m planning to add a new one: packages_cache_clear

freshports.dev=# INSERT INTO listen_for (name, script_name) values ('packages_cache_clear', 'ClearPackagesCache');
INSERT 0 1

EDIT 2020-05-01 : This was implemented using zfs rollback for cache clearing.

Found new repo release – automatically imported it

 packages, PostgreSQL, Stored procedures  No Responses »
Apr 282020
 

Today, while I wasn’t looking, both dev and test located and imported new package information for:

  • FreeBSD:11:amd64
  • FreeBSD:11:i386
  • FreeBSD:12:amd64
  • FreeBSD:12:i386
  • FreeBSD:13:amd64
  • FreeBSD:13:i386

However, the packages table was not correctly updated.

Why? Because I lost some function code I wrote.

This is the code which was in use, as mentioned in Using DEFERRABLE INITIALLY DEFERRED on constraints:

    UPDATE packages_raw PR
       SET abi_id  = abi.id,
           port_id = P.id
      FROM abi, ports P, element_pathname EP
     WHERE PR.abi         = 'FreeBSD:13:aarch64'
       AND PR.package_set = 'latest'
       AND PR.abi         = abi.name
       AND EP.pathname    = '/ports/head/' || PR.package_origin
       AND P.element_id   = EP.element_id;

As you can see, this will only update the FreeBSD:13:aarch64 ABI, for both latest and quarterly repos.

I needed the parameter driven version. Sadly, the above was in the repo, as expected. Looking back through the change log, it had never been saved.

I started thinking about backups I had taken of the database. As I was getting ready to restore old versions, I thought about where I might have saved this away during the work-in-progress phase.

I found this in an old explain:

 UPDATE packages_raw PR
       SET abi_id  = abi.id,
           port_id = PO.port_id
      FROM abi, ports_origin PO, ports P, element_pathname EP
     WHERE PR.abi         = 'FreeBSD:13:aarch64'
       AND PR.package_set = 'latest'
       AND PR.abi         = abi.name
       AND PO.port_origin = PR.package_origin
       AND PO.port_id     = P.id
       AND P.element_id   = EP.element_id
       AND EP.pathname like '/ports/head/%';

I remembered removing the ports_origin table from the query. This is what I massaged the above into:

    -- set abi_id and port_id before updating the pacakges table.
    UPDATE packages_raw PR
       SET abi_id  = abi.id,
           port_id = P.id
      FROM abi, ports P, element_pathname EP
     WHERE abi.id         = l_abi_id
       AND abi.name       = PR.abi
       AND PR.package_set = a_package_set
       AND EP.pathname    = '/ports/head/' || PR.package_origin
       AND P.element_id   = EP.element_id;

While hunting around old auto-saved TextEdit files, I found something which was identical to the above.

Here’s the final diff:

[dan@pg02:~/src/freshports/database-schema] $ svn di
Index: sp.txt
===================================================================
--- sp.txt	(revision 5312)
+++ sp.txt	(working copy)
@@ -3715,9 +3715,9 @@
        SET abi_id  = abi.id,
            port_id = P.id
       FROM abi, ports P, element_pathname EP
-     WHERE PR.abi         = 'FreeBSD:13:aarch64'
-       AND PR.package_set = 'latest'
-       AND PR.abi         = abi.name
+     WHERE abi.id         = l_abi_id
+       AND abi.name       = PR.abi
+       AND PR.package_set = a_package_set
        AND EP.pathname    = '/ports/head/' || PR.package_origin
        AND P.element_id   = EP.element_id;
 
[dan@pg02:~/src/freshports/database-schema] $

But wait, there’s more

While proof-reading this, I realized that I can remove abi from the above query.

We already have l_abi_id (line 13) and abi.name (line 12). Let’s try this instead:

    UPDATE packages_raw PR
       SET abi_id  = l_abi_id,
           port_id = P.id
      FROM ports P, element_pathname EP
     WHERE PR.abi          = a_abi_name
       AND PR.package_set = a_package_set
       AND EP.pathname    = '/ports/head/' || PR.package_origin
       AND P.element_id   = EP.element_id;

The explain analyse for this query is still good.

What’s next?

The packages cache needs to be automatically cleared after importing. At present, that’s a manual command:

[dan@test-nginx01:~] $ sudo rm -rf ~freshports/cache/packages/*
[dan@test-nginx01:~] $

The UpdatePackagesFromRawPackages.py script should raise a new signal.

No, it can’t be a signal. Signals are only within a given host. I has to be through the database because that’s the only real communication between the ingress host and the website.

Todays’s lessons

When you’ve got something good, make sure it’s saved to the repo.

If packages and ports are cached separately, how is one included in the middle of the other?

 Caching, packages  2 Responses »
Apr 182020
 

A new FreshPorts features is being developed. I’m trying to determine if packages for a port can be cached separately from the port itself. I will explain.

When you visit a port page, if it’s not cached, the page is built and then cached. Not the whole page, just the parts pertaining to that port. The right hand column is not cached.

If you look at https://dev.freshports.org/editors/joe you’ll see a table listing the available packages, as shown here:

Packages for joe

Packages for joe

At present, this table is cached with the rest of the port information.

My thinking: ports and packages are updated independently. It would be nice/cool/pretty damn eloquent to cache them separately.

PROBLEM: The ports cache is static HTML. How do you toss another bit of static HTML into the middle of that?

How caching works

When an update to a port occurs, the cache for that port is cleared. The next web request for that port will repopulate that cache.

While generating the HTML for that port, if the package HTML is not found, it can also be generated.

However, packages and port are updated via separate mechanisms:

  • port – via an incoming commit
  • packages – a new package repo build appears

PROBLEM: the packages cache can disappear right after you look to see if it’s there; it’s a race condition.

Conclusion

It will be far easier to cache the packages for a port with the port itself. To do otherwise creates a mess of problems to solve.

This means each time a new repo is build, the entire ports cache must be cleared. That’s just a bunch of files on disk. mv * ../DELETEME/ && rm -rf ../DELETEME/ will fix that.

Any other ideas

Do you have any ideas? All welcome. Thank you.

 Older Entries  Newer Entries