Today, kwm mentioned that http://www.freshports.org/Mk/bsd.port.mk was out of date, showing the most recent commit as back in March. My first concern was the system had stopped recording such commits. To check on that, I looked at https://svnweb.freebsd.org/ports/head/Mk/bsd.port.mk and searched my email for the most recent commit message. Extracting the message id from that email, I quickly found that FreshPorts was will processing such commits.
Checking the dev server I found that it had all the most recent commits.
This led me to thinking there was code which had not made it into production.
A bit of debugging had me thinking it was outdated SQL which was missing, but then I noticed it was a caching problem.
I cleared the cache on production and everything was appearing correctly.
To get a sample of what was going wrong, I took a recent commit and tested with it. I viewed the following pages to make sure they were cached:
- http://dev.freshports.org/net-mgmt/nagios-check_dhcp.pl/
- http://dev.freshports.org/net-mgmt/nagios-check_dhcp.pl/Makefile
At this point the cache contained this:
. ./head ./head/net-mgmt ./head/net-mgmt/nagios-check_dhcp.pl ./head/net-mgmt/nagios-check_dhcp.pl/Makefile.PageSize100.PageNum1.html ./net-mgmt ./net-mgmt/nagios-check_dhcp.pl ./net-mgmt/nagios-check_dhcp.pl/Detail.PageSize100.PageNum1.html
Why does the first entry have head in the pathname? Because it’s a non-port. That’s the crux of the issue I think.
To test the cache invalidation, I deleted the commit from the database:
freshports.org=# begin; BEGIN freshports.org=# delete from commit_log where message_id = '201410052140.s95LeUa9030114@svn.freebsd.org'; DELETE 1 freshports.org=# commit; COMMIT freshports.org=#
Then I reran the commit through the system. Here’s the cache after that:
. ./head ./head/Mk.PageSize100.PageNum1.html ./head/net-mgmt ./head/net-mgmt/nagios-check_dhcp.pl ./head/net-mgmt/nagios-check_dhcp.pl/Makefile.PageSize100.PageNum1.html ./net-mgmt ./net-mgmt/nagios-check_dhcp.pl
There’s the issue. The entry in head is still present.
Examination of the fp-listen logs shows:
Oct 5 22:24:07 webserver fp-listen: Just woke up! ************* Oct 5 22:24:07 webserver fp-listen: got port_updated and I need to call listen_port Oct 5 22:24:07 webserver fp-listen: checking for cache entries to remove... Oct 5 22:24:07 webserver fp-listen: COUNT: 1 entries to process Oct 5 22:24:07 webserver fp-listen: removing glob /usr/websites/freshports.org/dynamic/caching/cache/ports/net-mgmt/nagios-check_dhcp.pl/* Oct 5 22:24:07 webserver fp-listen: DELETE FROM cache_clearing_ports WHERE id = 1827187 Oct 5 22:24:07 webserver fp-listen: finished
It seems like I need to change everything in cache to be branch-aware. fp-listen will also need to allow for a branch, which means updating the cache_clearing_ports table to be branch-aware as well.
freshports.org=# \d cache_clearing_ports
Table "public.cache_clearing_ports"
Column | Type | Modifiers
------------+-----------------------------+-------------------------------------------------------------------
id | integer | not null default nextval('cache_clearing_ports_id_seq'::regclass)
port_id | integer | not null
category | text | not null
port | text | not null
date_added | timestamp without time zone | not null default now()
Indexes:
"cache_clearing_ports_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"cache_clearing_ports_port_id_fkey" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE CASCADE ON DELETE CASCADE
Ironically, I’ve discovered that the website caches the page for Mk/bsd.port.mk but never invalidates it, as found in the processing logs for a recent commit:
done RecordPortsTreeButNonPortCommits No errors found during that commit Observer has noticed that the update for 201410041805.s94I5nHf088546@svn.freebsd.org has finished. Observer will clear the following ports from cache after the commit: Observer will clear the following files from cache after the commit: we are ignoring /ports/head/Mk/bsd.port.mk for cache clearing *** end of items to be cleared
That clearly must change.
