How FreshPorts processes vuxml entries

One of my favorite FreeBSD features is security/vuxml – from it we get these great tools (provided by pkg):

* pkg audit (what installed packages contain known vulnerabilities?)
* /usr/local/etc/periodic/security/410.pkg-audit – pkg-audit for host and jails
* /usr/local/etc/periodic/security/405.pkg-base-audit – same as above, but for base system

I use the latter two command within my monitoring system (Nagios). This pkg-base-audit.sh example can be easily adapted by you for pkg-audit.

In this post:

FreeBSD 14.2
perl 5.42
How does FreshPorts process vuxml entries – this is a lead-in to modifying the process to cater for deleted entries – see also rectracted vuxml entry still in freshports database

With that stated, let’s start with the life cycle of how the process runs.

The commit

The details of FreshPorts commit processing is outside scope. However, we will cover the part which pertains to vuxml updates. There are certain files which are flagged for special handling after the commit is processed. That is implemented in modules/special_processing_files.pm – at the time of writing those special files were:

$FreshPorts::Constants::PORTS_MOVED – /ports/head/MOVED
$FreshPorts::Constants::PORTS_UPDATING – /ports/head/UPDATING
$FreshPorts::Constants::VUXML – \/ports\/head\/security\/vuxml/vuln/(\d{4})?.xml – i.e. all the vuxml vuln database files, sorted by year
$FreshPorts::Constants::DEFAULT_VERSION – /ports/head/Mk/bsd.default-versions.mk

When the right file is encountered, the following flags are set:

                `/usr/bin/touch $FreshPorts::Config::VuXMLFileFlag`;
                `/usr/bin/touch $FreshPorts::Config::JobWaiting`;

Those signal files are located in ~freshports/signals/

A typical set of processes

This is a typical set of processes running in an otherwise-idle FreshPorts ingress node:

[11:58 dev-ingress01 dvl ~/scripts] % ps auwwx | grep -v dvl
USER         PID %CPU %MEM   VSZ   RSS TT  STAT STARTED    TIME COMMAND
nagios      7821  0.0  0.0 19596  7728  -  IsJ   6Jul25 0:44.58 /usr/local/sbin/nrpe -c /usr/local/etc/nrpe.cfg -d
root       16842  0.0  0.0 13904  2724  -  SsJ   4Jul25 0:37.54 /usr/sbin/syslogd -c -ss
root       16968  0.0  0.0 23864 10060  -  IsJ   4Jul25 0:07.12 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
root       16994  0.0  0.0 13944  2488  -  IsJ   4Jul25 0:11.34 /usr/sbin/cron -s
ingress    40580  0.0  0.0 13856  2308  -  IsJ  21Jul25 0:00.60 daemon: ingress[40581] (daemon)
ingress    40581  0.0  0.0 14408  2908  -  SJ   21Jul25 0:30.01 /bin/sh /usr/local/libexec/freshports-service/ingress.sh
freshports 40603  0.0  0.0 13856  2312  -  IsJ  21Jul25 0:00.27 daemon: freshports[40604] (daemon)
freshports 40604  0.0  0.0 14408  3272  -  SJ   21Jul25 3:06.95 /bin/sh /usr/local/libexec/freshports-service/freshports.sh
freshports 94259  0.0  0.0 13740  2144  -  SCJ  11:58   0:00.00 sleep 3
ingress    94261  0.0  0.0 13740  2144  -  SCJ  11:58   0:00.00 sleep 3

The two main processes are ingress (pulling commits out of git and into this node) and freshports (processing those commits into the database).

Detecting the flag

When the freshports process detects the JobWaiting flag, it launches job-waiting.pl. This script is launched by both daemons (ingress and freshports). It decides what it can do based upon what user is running it.

The script associated with the VuXMLFileFlag flag/signal is process_vuxml.sh

job_waiting.pl is a simple script. It looks to see what flags are set and launches a script – no parameters, nothing fancy – it just launches the indicated script.

VuXMLFileFlag

process_vuxml.sh has two main tasks:

/usr/local/bin/perl ./process_vuxml.pl –filename=${VULNFILE} –showreasons >> ${LOGFILE} – refreshes the database
/usr/local/bin/perl ./vuln_latest.pl >> ${LOGFILE} – updates the HTML for the Latest Vulnerabilities box at the tope right of the webpage

We will concentrate on process_vuxml.pl.

process_vuxml.pl

The main job of this script, as taken from the comments:

# Split up the vuln.xml file into sections for individual
# vulnerabilities.  Save into files using the vid guid field as name.
# Calculate SHA256 checksum for the XML snippet and write out to an
# index file.

The code is set up to process each entry in the vuxml database provided by security/vuxml.

What it needs to do is check for any entries in the FreshPorts database which is not found in the vuxml database.

Idea

To do this simply, the code would:

Grab the list of vid values from the vuxml table in the FreshPorts database
Store that list in a perl hash/array
As each vuln from the security/vuxml database is processed, remove that vid from the hash
When processing is completed, any entry remaining in the hash can be deleted from the FreshPorts database.

But wait! There’s more!

While typing the previous section, I looked at the database schema. I found:

[12:15 pg03 dvl ~] % psql freshports.dev
psql (16.9)
Type "help" for help.

freshports.dev=# \d vuxml
                                   Table "public.vuxml"
     Column     |     Type     | Collation | Nullable |              Default              
----------------+--------------+-----------+----------+-----------------------------------
 id             | integer      |           | not null | nextval('vuxml_id_seq'::regclass)
 vid            | text         |           | not null | 
 topic          | text         |           | not null | 
 description    | text         |           | not null | 
 date_discovery | text         |           |          | 
 date_entry     | text         |           |          | 
 date_modified  | text         |           |          | 
 status         | character(1) |           | not null | 
 checksum       | text         |           |          | 
Indexes:
    "vuxml_pkey" PRIMARY KEY, btree (id)
    "vuxml_vid_idx" btree (vid)
Referenced by:
    TABLE "vuxml_affected" CONSTRAINT "$1" FOREIGN KEY (vuxml_id) REFERENCES vuxml(id) ON UPDATE CASCADE ON DELETE CASCADE
    TABLE "vuxml_references" CONSTRAINT "$1" FOREIGN KEY (vuxml_id) REFERENCES vuxml(id) ON UPDATE CASCADE ON DELETE CASCADE
    TABLE "commit_log_ports_vuxml" CONSTRAINT "$1" FOREIGN KEY (vuxml_id) REFERENCES vuxml(id) ON UPDATE CASCADE ON DELETE CASCADE

freshports.dev=# select distinct status from vuxml;
 status 
--------
 A
(1 row)

freshports.dev=# select count(*) from vuxml;
 count 
-------
  6061
(1 row)

freshports.dev=#

See that status field on line 16? How about setting that to ‘D’ at the start of vuxml processing?

As each entry is updated, we reset status to ‘A’. At the end, delete everything with ‘D’.

That’s very simple.

However, it bumps your database IO from 1 to 6000 updates – every row in that table is updated every time the vuxml files are processed.

Initially, I was tempted by the above solution, now, the perl hash solution seems better.

I’m sure this status field is otherwise unused and exists for future use.

A temp table

What we could also do: load all the vid values from the vuxml files into a temporary table. Then delete from the vuxml table any vid values which do not appear in the temporary table.

This would not need 6000 inserts into the temp table – COPY could be used instead. If we also inserted the checksums into this table, we could use SQL to determine the changed entries, instead of doing 6000 selects (which is what the code does now).

I think this solution may be our best option. Always let the database do the work.