Oct 052014

Today, kwm mentioned that http://www.freshports.org/Mk/bsd.port.mk was out of date, showing the most recent commit as back in March. My first concern was the system had stopped recording such commits. To check on that, I looked at https://svnweb.freebsd.org/ports/head/Mk/bsd.port.mk and searched my email for the most recent commit message. Extracting the message id from that email, I quickly found that FreshPorts was will processing such commits.

Checking the dev server I found that it had all the most recent commits.

This led me to thinking there was code which had not made it into production.

A bit of debugging had me thinking it was outdated SQL which was missing, but then I noticed it was a caching problem.

I cleared the cache on production and everything was appearing correctly.

To get a sample of what was going wrong, I took a recent commit and tested with it. I viewed the following pages to make sure they were cached:

  1. http://dev.freshports.org/net-mgmt/nagios-check_dhcp.pl/
  2. http://dev.freshports.org/net-mgmt/nagios-check_dhcp.pl/Makefile

At this point the cache contained this:


Why does the first entry have head in the pathname? Because it’s a non-port. That’s the crux of the issue I think.

To test the cache invalidation, I deleted the commit from the database:

freshports.org=# begin;
freshports.org=# delete from commit_log where message_id = '201410052140.s95LeUa9030114@svn.freebsd.org';
freshports.org=# commit;

Then I reran the commit through the system. Here’s the cache after that:


There’s the issue. The entry in head is still present.

Examination of the fp-listen logs shows:

Oct  5 22:24:07 webserver fp-listen: Just woke up! *************
Oct  5 22:24:07 webserver fp-listen: got port_updated and I need to call listen_port
Oct  5 22:24:07 webserver fp-listen: checking for cache entries to remove...
Oct  5 22:24:07 webserver fp-listen: COUNT: 1 entries to process
Oct  5 22:24:07 webserver fp-listen: removing glob /usr/websites/freshports.org/dynamic/caching/cache/ports/net-mgmt/nagios-check_dhcp.pl/*
Oct  5 22:24:07 webserver fp-listen: DELETE FROM cache_clearing_ports WHERE id = 1827187
Oct  5 22:24:07 webserver fp-listen: finished

It seems like I need to change everything in cache to be branch-aware. fp-listen will also need to allow for a branch, which means updating the cache_clearing_ports table to be branch-aware as well.

freshports.org=# \d cache_clearing_ports
                                     Table "public.cache_clearing_ports"
   Column   |            Type             |                             Modifiers                             
 id         | integer                     | not null default nextval('cache_clearing_ports_id_seq'::regclass)
 port_id    | integer                     | not null
 category   | text                        | not null
 port       | text                        | not null
 date_added | timestamp without time zone | not null default now()
    "cache_clearing_ports_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
    "cache_clearing_ports_port_id_fkey" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE CASCADE ON DELETE CASCADE

Ironically, I’ve discovered that the website caches the page for Mk/bsd.port.mk but never invalidates it, as found in the processing logs for a recent commit:

done RecordPortsTreeButNonPortCommits
No errors found during that commit
Observer has noticed that the update for 201410041805.s94I5nHf088546@svn.freebsd.org has finished.
Observer will clear the following ports from cache after the commit:
Observer will clear the following files from cache after the commit:
we are ignoring /ports/head/Mk/bsd.port.mk for cache clearing
*** end of items to be cleared

That clearly must change.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive