Jun 212021
 

FreshPorts has three main daemons:

  1. ingress – waits for incoming commits and creates XML.
  2. freshports – processes the XML into the database.
  3. fp-listen – listens for events and clears cache

The first two run on the backend (the ingress node, or jail), the last run runs on the webserver (nginx jail).

The list of events

The first thing fp-listen does is read the list of events from the database:

freshports.devgit=# select * from listen_for;
 id |       name        |       script_name        
----+-------------------+--------------------------
  5 | port_updated      | listen_port
  6 | ports_moved       | listen_ports_moved
  7 | ports_updating    | listen_ports_updating
  8 | vuxml             | listen_vuxml
 10 | category_new      | listen_category_new
 11 | date_updated      | listen_date_updated
 12 | packages_imported | listen_packages_imported
 13 | listen_for        | ClearPackagesCache
(8 rows)

Let’s add our new event and the name of the script we-have-yet-to-write:

freshports.devgit=# insert into listen_for (name, script_name) values ('commit_updated', 'listen_commit');
INSERT 0 1

However, note that I won’t actually be created a script by that name.

The Python code

Here is the Python code:

[dan@devgit-nginx01:~/src/fp-listen/fp-listen] $ svn di
Index: fp-listen.py
===================================================================
--- fp-listen.py	(revision 5642)
+++ fp-listen.py	(working copy)
@@ -110,6 +110,44 @@
 
   syslog.syslog(syslog.LOG_NOTICE, "Done with PackagesCacheClear()");
 
+def CommitsCacheClear():
+  dbh = psycopg2.connect(DSN)
+  curs = dbh.cursor(cursor_factory=psycopg2.extras.DictCursor)
+
+  curs.execute("SELECT commit_to_clear FROM cache_clearing_commits")
+  NumRows = curs.rowcount
+  dbh.commit();
+  if (NumRows > 0):
+    syslog.syslog(syslog.LOG_NOTICE, 'COUNT: %d entries to process' % (NumRows))
+    rows = curs.fetchall()
+    for row in rows:
+      #
+      filenameglob = config['dirs']['COMMIT_CACHE_PATH'] % (row['commit_to_clear'])
+      syslog.syslog(syslog.LOG_NOTICE, 'removing glob %s' % (filenameglob))
+
+      try:
+        for filename in glob.glob(filenameglob):
+          syslog.syslog(syslog.LOG_NOTICE, 'removing %s' % (filename))
+          if os.path.isfile(filename):
+            os.remove(filename)
+          else:
+            shutil.rmtree(filename)
+
+      except FileNotFoundError:
+        syslog.syslog(syslog.LOG_CRIT, 'could not find file for deletion %s' % (filename))
+
+      syslog.syslog(syslog.LOG_NOTICE, "DELETE FROM cache_clearing_commits WHERE commit_to_clear = '%s'" % (row['commit_to_clear']))
+      curs.execute("DELETE FROM cache_clearing_commits WHERE commit_to_clear = '%s'" % (row['commit_to_clear']))
+      dbh.commit()
+
+    # end for
+  else:
+    syslog.syslog(syslog.LOG_ERR, 'ERROR: No cached entries found for removal')
+  # end if
+    
+  syslog.syslog(syslog.LOG_NOTICE, 'finished')
+  return NumRows
+
 def ClearDateCacheEntries():
   syslog.syslog(syslog.LOG_NOTICE, 'checking for cache date to remove...')
   dbh = psycopg2.connect(DSN)
@@ -154,6 +192,7 @@
   syslog.syslog(syslog.LOG_NOTICE, 'finished')
   return NumRows
 
+
 def Touch(File):
   if not os.path.exists(File):
     fd = open(File, 'aos.O_WRONLY | os.O_NONBLOCK | os.O_CREAT | os.O_NOCTTY | os.O_APPEND')
@@ -160,6 +199,7 @@
     fd.close()
   os.utime(File, None)
 
+
 def ProcessCategoryNew():
   syslog.syslog(syslog.LOG_NOTICE, 'We have a new category')
   
@@ -167,14 +207,18 @@
   Touch(config['flags']['WWWENPortsCategoriesFlag'])
   Touch(config['flags']['JOBWAITING'])
 
+
 def ProcessPortsMoved():
   syslog.syslog(syslog.LOG_NOTICE, 'processing ports/MOVED')
 
+
 def ProcessPortsUpdating():
   syslog.syslog(syslog.LOG_NOTICE, 'processing ports/UPDATING')
 
+
 def ProcessVUXML():
   syslog.syslog(syslog.LOG_NOTICE, 'processing ports/security/portaudit/vuln.xml')
+
   
 def ClearMiscCaches():
   syslog.syslog(syslog.LOG_NOTICE, 'invoked: ClearMiscCaches()');
@@ -261,6 +305,9 @@
           # at the time of writing, there was no reason to ClearMiscCaches() when
           # new packages arrive
           clear_cache = False;
+        elif listens[notify.channel] == 'listen_commit':
+          syslog.syslog(syslog.LOG_NOTICE, "invoking CommitsClearCache()");
+          CommitsCacheClear()
         else:
           clear_cache = False;
           syslog.syslog(syslog.LOG_ERR, "Code does not know what to do when '%s' is found." % notify.channel)
[dan@devgit-nginx01:~/src/fp-listen/fp-listen] $ 

Directory permissions

When doing the initial testing, I saw this error:

Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: Traceback (most recent call last):
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:   File "/usr/local/lib/python3.8/site-packages/fp-listen/fp-listen.py", line 310, in 
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:     CommitsCacheClear()
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:   File "/usr/local/lib/python3.8/site-packages/fp-listen/fp-listen.py", line 134, in CommitsCacheClear
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:     shutil.rmtree(filename)
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:   File "/usr/local/lib/python3.8/shutil.py", line 722, in rmtree
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:     onerror(os.rmdir, path, sys.exc_info())
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:   File "/usr/local/lib/python3.8/shutil.py", line 720, in rmtree
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]:     os.rmdir(path)
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: PermissionError: [Errno 13] Permission denied: '/var/db/freshports/cache/commits/400c1af36e44f6ad699125526c70668745482c98/'

Ahh, that’s because we have this:

[dan@devgit-nginx01:/usr/local/etc/freshports] $ ls -l /var/db/freshports/cache
total 432
drwxr-xr-x  2 www         freshports       2 Feb 28 18:20 categories
drwxr-xr-x  3 www         freshports       3 Jun 21 23:54 commits
drwxrwxr-x  3 www         freshports       3 Jun 20 15:51 daily
drwxr-xr-x  2 www         freshports       2 Feb 28 18:20 general
drwxrwxr-x  2 freshports  freshports       9 Jun 21 23:00 html
drwxrwxr-x  2 www         freshports       2 Jun 21 23:47 news
drwxrwxr-x  2 www         freshports       2 Feb 28 18:20 packages
drwxr-xr-x  2 www         freshports       2 Feb 28 18:20 pages
drwxrwxr-x  6 www         freshports       6 Jun 20 14:30 ports
-rw-r--r--  1 www         freshports  414331 Jun 13 20:41 searchlog.txt
drwxr-xr-x  2 www         freshports       3 Jun 21 23:54 spooling

Ahh, freshports can’t write to the commits directory, so it cannot delete files. The fp-listen daemon is running as the freshports user:

$ ps auwwx | grep fp-listen
freshports 44828  0.0  0.0  29756 19472  -  SJ   00:06   0:00.19 /usr/local/bin/python /usr/local/lib/python3.8/site-packages/fp-listen/fp-listen.py (python3.8)
dan        77297  0.0  0.0  11456  2776  6  S+J  00:22   0:00.00 grep fp-listen

I can’t just chmod +w here. There is a zfs snapshot involved. Here is what I did instead:

$ sudo zfs rollback nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits@empty
$ sudo zfs destroy nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits@empty
$ sudo chmod g+w commits
$ sudo zfs snapshot nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits@empty

The destroy had to be done on the host itself. This is a situation I’ve seen before where a snapshot cannot be deleted from within the jail, but it can be created.

Configuration file changes

This new entry is required in /usr/local/etc/freshports/fp-listen.ini:

[dan@devgit-nginx01:~/src/fp-listen] $ svn di etc/freshports/fp-listen.ini.sample
Index: etc/freshports/fp-listen.ini.sample
===================================================================
--- etc/freshports/fp-listen.ini.sample	(revision 5642)
+++ etc/freshports/fp-listen.ini.sample	(working copy)
@@ -13,6 +13,7 @@
 BASEDIR         = /var/db/freshports
 
 CATEGORY_CACHE_PATH = %(BASEDIR)s/cache/categories/%%s/*
+COMMIT_CACHE_PATH   = %(BASEDIR)s/cache/commits/%%s/
 PORT_CACHE_PATH     = %(BASEDIR)s/cache/ports/%%s/%%s/*
 DATE_CACHE_PATH     = %(BASEDIR)s/cache/daily/%%s/%%s/%%s.*
 NEWS_CACHE_DIR      = %(BASEDIR)s/cache/news/
[dan@devgit-nginx01:~/src/fp-listen] $ 
Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive