Today at about 1:00 pm Philadelphia time, the website went offline. It was still there, it just could not answer any queries. In this post: Website and database server running in AWS FreeBSD 12.2 on a t3.medium EC2 instance PostgreSQL 12.5 running on a db.t2.large RDS instance Initial look Looking at load, it was acceptable: […]
lack of index brings website to a crawl Read More »
This post is a how-to and reminder for myself. I’m working on finding a better way for FreshPorts to know a new physical ports category when it finds it. I think the only way is to look at the directories in the FreeBSD ports repo. A discussion on IRC led to this shell script: [dan@devgit-ingress01:/var/db/ingress/repos/ports]
Generating XML for a single commit Read More »
In this post I’ll talk about how the code intially took the easy way out and how recent changes have tightened that up. In short, the code would raise N messages when just a single notification is sufficient. Today the full vuxml database was reloaded into FreshPorts. This affects approximately 3454 ports and the marking
Changing how we record ports which need cache clearing Read More »
A recent post on the FreeBSD Ports mailing list asked: Hi, I noticed a big difference between the number of ports on freebsd.org/ports/ and on freshports.org. Currently, it’s 33348 vs. 41346. The freebsd.org’s number equals roughly the number of lines of a current INDEX, but how does FreshPorts count? Best, Moritz In short, they are
How many ports are there? Read More »
Today I’m working on https://devgit.freshports.org and fixing links to http://github.com/freebsd/freebsd-ports/ for individual commits. Some of these links are stunted to: commit/c2b0677 (i.e. no hostname) A theory I think I know why. The repo_id field of the commit_log table is empty. This field links to the commit to a specific row in the repo table. That
Fixing links to git repo Read More »
I noticed this today: root 58697 0.0 0.0 10680 2180 – IJ 12May20 0:00.00 /usr/local/bin/readproctitle service errors: …/site_perl/FreshPorts/vuxml_parsing.pm line 232, chunk 1.\nWide character in print at /usr/local/lib/perl5/site_perl/FreshPorts/vuxml_parsing.pm line 232, chunk 1.\nWide character in print at /usr/local/lib/perl5/site_perl/FreshPorts/vuxml_parsing.pm line 232, chunk 1.\nUndefined subroutine &FreshPorts::CommitterOptIn::RecordErrorDetails called at ./process_vuxml.pl line 124, chunk 1.\n I should monitor that process better.
FreshPorts had a cache clearing issue recently. It lingered. For a long time. It took me a while to figure it out. It turned out to be a Python 3 compatibility issue. In this post: PostgreSQL 12.1 FreeBSD 12.1 I will outline how database changes invoke external scripts asynchronously which then clear the cache. I
Cache clearing issue Read More »
I’ve noticed a log of sanity test failures related to devel/binutils lately. Every single commit. Issue 133 deals with that. Here, I plan to document what happened. FreshPorts has, since inception, worked off the commit emails. The incoming email is parsed and XML created. The XML is then used to update the database. I started
The slave ports for devel/binutils did not get deleted Read More »
XML can be tricky. XML plays a heavy role in FreshPorts. Since the early days, converting to XML then loading into the database has been a priority. This choice means the database-loading code doesn’t need to know much about the source of the data. It always means we can change the data source without modifying
Code point \u0005 is not a valid character in XML Read More »
I’m writing this post just to keep things straight in my head so I can decide how best to resolve this issue. FreshPorts uses /var/db/freshports/cache/spooling on both the ingress jail and the nginx jail. The nginx jail uses it for caching content. Page details are first spooled into /var/db/freshports/cache/spooling before moving it to /var/db/freshports/cache/ports. The
I have a spooling directory issue to fix Read More »