FreshPorts has three main daemons:
- ingress – waits for incoming commits and creates XML.
- freshports – processes the XML into the database.
- fp-listen – listens for events and clears cache
The first two run on the backend (the ingress node, or jail), the last run runs on the webserver (nginx jail).
The list of events
The first thing fp-listen does is read the list of events from the database:
freshports.devgit=# select * from listen_for; id | name | script_name ----+-------------------+-------------------------- 5 | port_updated | listen_port 6 | ports_moved | listen_ports_moved 7 | ports_updating | listen_ports_updating 8 | vuxml | listen_vuxml 10 | category_new | listen_category_new 11 | date_updated | listen_date_updated 12 | packages_imported | listen_packages_imported 13 | listen_for | ClearPackagesCache (8 rows)
Let’s add our new event and the name of the script we-have-yet-to-write:
freshports.devgit=# insert into listen_for (name, script_name) values ('commit_updated', 'listen_commit');
INSERT 0 1
However, note that I won’t actually be created a script by that name.
The Python code
Here is the Python code:
[dan@devgit-nginx01:~/src/fp-listen/fp-listen] $ svn di
Index: fp-listen.py
===================================================================
--- fp-listen.py (revision 5642)
+++ fp-listen.py (working copy)
@@ -110,6 +110,44 @@
syslog.syslog(syslog.LOG_NOTICE, "Done with PackagesCacheClear()");
+def CommitsCacheClear():
+ dbh = psycopg2.connect(DSN)
+ curs = dbh.cursor(cursor_factory=psycopg2.extras.DictCursor)
+
+ curs.execute("SELECT commit_to_clear FROM cache_clearing_commits")
+ NumRows = curs.rowcount
+ dbh.commit();
+ if (NumRows > 0):
+ syslog.syslog(syslog.LOG_NOTICE, 'COUNT: %d entries to process' % (NumRows))
+ rows = curs.fetchall()
+ for row in rows:
+ #
+ filenameglob = config['dirs']['COMMIT_CACHE_PATH'] % (row['commit_to_clear'])
+ syslog.syslog(syslog.LOG_NOTICE, 'removing glob %s' % (filenameglob))
+
+ try:
+ for filename in glob.glob(filenameglob):
+ syslog.syslog(syslog.LOG_NOTICE, 'removing %s' % (filename))
+ if os.path.isfile(filename):
+ os.remove(filename)
+ else:
+ shutil.rmtree(filename)
+
+ except FileNotFoundError:
+ syslog.syslog(syslog.LOG_CRIT, 'could not find file for deletion %s' % (filename))
+
+ syslog.syslog(syslog.LOG_NOTICE, "DELETE FROM cache_clearing_commits WHERE commit_to_clear = '%s'" % (row['commit_to_clear']))
+ curs.execute("DELETE FROM cache_clearing_commits WHERE commit_to_clear = '%s'" % (row['commit_to_clear']))
+ dbh.commit()
+
+ # end for
+ else:
+ syslog.syslog(syslog.LOG_ERR, 'ERROR: No cached entries found for removal')
+ # end if
+
+ syslog.syslog(syslog.LOG_NOTICE, 'finished')
+ return NumRows
+
def ClearDateCacheEntries():
syslog.syslog(syslog.LOG_NOTICE, 'checking for cache date to remove...')
dbh = psycopg2.connect(DSN)
@@ -154,6 +192,7 @@
syslog.syslog(syslog.LOG_NOTICE, 'finished')
return NumRows
+
def Touch(File):
if not os.path.exists(File):
fd = open(File, 'aos.O_WRONLY | os.O_NONBLOCK | os.O_CREAT | os.O_NOCTTY | os.O_APPEND')
@@ -160,6 +199,7 @@
fd.close()
os.utime(File, None)
+
def ProcessCategoryNew():
syslog.syslog(syslog.LOG_NOTICE, 'We have a new category')
@@ -167,14 +207,18 @@
Touch(config['flags']['WWWENPortsCategoriesFlag'])
Touch(config['flags']['JOBWAITING'])
+
def ProcessPortsMoved():
syslog.syslog(syslog.LOG_NOTICE, 'processing ports/MOVED')
+
def ProcessPortsUpdating():
syslog.syslog(syslog.LOG_NOTICE, 'processing ports/UPDATING')
+
def ProcessVUXML():
syslog.syslog(syslog.LOG_NOTICE, 'processing ports/security/portaudit/vuln.xml')
+
def ClearMiscCaches():
syslog.syslog(syslog.LOG_NOTICE, 'invoked: ClearMiscCaches()');
@@ -261,6 +305,9 @@
# at the time of writing, there was no reason to ClearMiscCaches() when
# new packages arrive
clear_cache = False;
+ elif listens[notify.channel] == 'listen_commit':
+ syslog.syslog(syslog.LOG_NOTICE, "invoking CommitsClearCache()");
+ CommitsCacheClear()
else:
clear_cache = False;
syslog.syslog(syslog.LOG_ERR, "Code does not know what to do when '%s' is found." % notify.channel)
[dan@devgit-nginx01:~/src/fp-listen/fp-listen] $
Directory permissions
When doing the initial testing, I saw this error:
Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: Traceback (most recent call last): Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: File "/usr/local/lib/python3.8/site-packages/fp-listen/fp-listen.py", line 310, inJun 21 23:56:17 devgit-nginx01 fp_listen[77170]: CommitsCacheClear() Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: File "/usr/local/lib/python3.8/site-packages/fp-listen/fp-listen.py", line 134, in CommitsCacheClear Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: shutil.rmtree(filename) Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: File "/usr/local/lib/python3.8/shutil.py", line 722, in rmtree Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: onerror(os.rmdir, path, sys.exc_info()) Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: File "/usr/local/lib/python3.8/shutil.py", line 720, in rmtree Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: os.rmdir(path) Jun 21 23:56:17 devgit-nginx01 fp_listen[77170]: PermissionError: [Errno 13] Permission denied: '/var/db/freshports/cache/commits/400c1af36e44f6ad699125526c70668745482c98/'
Ahh, that’s because we have this:
[dan@devgit-nginx01:/usr/local/etc/freshports] $ ls -l /var/db/freshports/cache total 432 drwxr-xr-x 2 www freshports 2 Feb 28 18:20 categories drwxr-xr-x 3 www freshports 3 Jun 21 23:54 commits drwxrwxr-x 3 www freshports 3 Jun 20 15:51 daily drwxr-xr-x 2 www freshports 2 Feb 28 18:20 general drwxrwxr-x 2 freshports freshports 9 Jun 21 23:00 html drwxrwxr-x 2 www freshports 2 Jun 21 23:47 news drwxrwxr-x 2 www freshports 2 Feb 28 18:20 packages drwxr-xr-x 2 www freshports 2 Feb 28 18:20 pages drwxrwxr-x 6 www freshports 6 Jun 20 14:30 ports -rw-r--r-- 1 www freshports 414331 Jun 13 20:41 searchlog.txt drwxr-xr-x 2 www freshports 3 Jun 21 23:54 spooling
Ahh, freshports can’t write to the commits directory, so it cannot delete files. The fp-listen daemon is running as the freshports user:
$ ps auwwx | grep fp-listen freshports 44828 0.0 0.0 29756 19472 - SJ 00:06 0:00.19 /usr/local/bin/python /usr/local/lib/python3.8/site-packages/fp-listen/fp-listen.py (python3.8) dan 77297 0.0 0.0 11456 2776 6 S+J 00:22 0:00.00 grep fp-listen
I can’t just chmod +w here. There is a zfs snapshot involved. Here is what I did instead:
$ sudo zfs rollback nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits@empty $ sudo zfs destroy nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits@empty $ sudo chmod g+w commits $ sudo zfs snapshot nvd/freshports/devgit-nginx01/var/db/freshports/cache/commits@empty
The destroy had to be done on the host itself. This is a situation I’ve seen before where a snapshot cannot be deleted from within the jail, but it can be created.
Configuration file changes
This new entry is required in /usr/local/etc/freshports/fp-listen.ini:
[dan@devgit-nginx01:~/src/fp-listen] $ svn di etc/freshports/fp-listen.ini.sample Index: etc/freshports/fp-listen.ini.sample =================================================================== --- etc/freshports/fp-listen.ini.sample (revision 5642) +++ etc/freshports/fp-listen.ini.sample (working copy) @@ -13,6 +13,7 @@ BASEDIR = /var/db/freshports CATEGORY_CACHE_PATH = %(BASEDIR)s/cache/categories/%%s/* +COMMIT_CACHE_PATH = %(BASEDIR)s/cache/commits/%%s/ PORT_CACHE_PATH = %(BASEDIR)s/cache/ports/%%s/%%s/* DATE_CACHE_PATH = %(BASEDIR)s/cache/daily/%%s/%%s/%%s.* NEWS_CACHE_DIR = %(BASEDIR)s/cache/news/ [dan@devgit-nginx01:~/src/fp-listen] $
