Oct 102017
 

The previous post on this blog was nearly 10 months ago. However, things have not stood still. There have been a few issues fixed and some are still in progress.

The biggest thing underway is a major rework of the backend, the part that processes the incoming commits. There is nothing major; no huge code rewrites.

The basics remain:

  • FreshPorts gets an email from one of the commit mailing list.
  • That email is converted to XML and saved to disk, in a queue.
  • The queue is processed, one commit at a time.
  • An svn up is run.
  • A bunch of make -V occur
  • The database is updated.
  • Profit.

In current production, all of this occurs on one host.

In the new system, I will be taking advantage / exploiting many features of FreeBSD which were not available when FreshPorts started.

The hardware changes

The main change is ZFS. The existing system is on a hardware RAID card with 8 drives. This system was deployed in mid-2006 and it has been a great worker.

The new system boots off a pair of SSDs on ZFS zroot. I plan to make use of beadm for upgrades.

Also present are two 5TB drives, also in a ZFS mirror.

The OS has been installed, but so has most of the FreshPorts code. Over the past few weeks I have been redesigning the back-end to cater for the use of multiple jails. Why jails? I want to separate the main components of the system. This makes it easier to upgrade individual components without affecting the others. It also increase security by isolating various components to their respective worlds.

Jails

There are four jails on this server:

  1. nginx01 – runs the website, and php-fpm, because the website is based on PHP
  2. pg01 – Run PostgreSQL, the database which stores all the good stuff. No public IP address. Accessed over lo1 via a 127.1.0.0/24 address.
  3. mx-ingress01 – accepts incoming email and passes it on to the next stage.
  4. ingress01 – accepts email from mx-ingress01 and performs the basic steps mentioned in the first section of this blog post. No public IP address.

Each of these jails have been created by hand, but the software and daemons have been configured via Ansible.

Porting the code makes some stuff easier

When it came time to deploy the backend FreshPorts code into ingress01, I created a port. This is a departure from current production where code is deployed via svn up.

With the use of a port, many other things came with it:

  • All the backend code goes into /usr/local/libexec/freshports<./li>
  • Main configuration files are now in /usr/local/etc/freshports with symlinks to there from /usr/local/libexec/freshports.
  • /var/db/freshports is the main location for any data external to the database on pg01.
  • /var/db/freshports/cache is the cache for the webpages, with multiple directories for different classes of pages.
  • /var/db/ingress/message-queues/incoming is where the incoming XML data arrives. The ingress user writes here, and the freshports user reads and then moves the file to another queue. All of this occurs in the ingress01 jail.
  • The freshports user has /var/db/freshports/message-queues/ for commit message storage (i.e. the XML). Subdirectories include archive, recent, and retry.
  • The ports-jail is at /var/db/freshports/ports-jail and is where the freshports user does a chroot before running any make -V commands.
  • Within the above named directory, we have /var/db/freshports/ports-jail/var/db/repos which is where the svn up commands are run.
  • Logging is consolidated. It used to all go to /var/log/messages. Now it goes to /var/log/freshports/ and into a number of files contained therein. The port also installed entries to /usr/local/etc/syslog.d/ and to /usr/local/etc/newsyslog.d/ to ensure logs are both captured and rotated.

All of these directories are created by the port. That creating reduces the configuration time. I had scripts, but they were each run manually. Now, it’s fully automated, complete with permission, and user creation.

All of this is handled by the freshports-scripts port.

The freshports daemon

There has long been a freshports deamon, a small python script which looks for incoming XML files and processes them. This is powered by daemontools.

I suppose it could be replaced by a rc.d script, but I might do that another day.

This service is handled by the newly-created freshports-services port.

Current status

As of now, commits are coming in and being processed automatically. The website is being updated and all seems well. There are no known problems.

Remaining work

Before this website goes live, a number of items remain to be completed.

  1. I am not sure that UPDATING and MOVED commits are being processed properly. EDIT 2017.10.14 – done
  2. The fp-listen daemon, which listens for completed commits and clears the appropriate cache bits, is not running. EDIT 2017.10.11 – done
  3. There are a number of cronjob tasks which I should verify are running properly. They always ran as my user id, and they need to run as the freshports user.
  4. I need to get the website running with https under nginx. EDIT 2017.10.14 –
    done

None of these are huge tasks, just time consuming.

Summary

All of these changes are for good. They were not done in the past because there was no reason for them. Now that I’m trying to split tasks up into different jails, the changes make sense and the work is being done.

These changes will also make the inevitable transition under the control of the FreeBSD project. That will not happen soon, but it will happen. I can’t run FreshPorts for ever.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive

  One Response to “Redesign of FreshPorts backend”

  1. Here are the directories on the webserver:

    $ find /var/db/freshports/
    /var/db/freshports/
    /var/db/freshports/ports-jail
    /var/db/freshports/ports-jail/lib
    /var/db/freshports/ports-jail/make-master-sites-all.sh
    /var/db/freshports/ports-jail/sbin
    /var/db/freshports/ports-jail/cat-descr.sh
    /var/db/freshports/ports-jail/bin
    /var/db/freshports/ports-jail/usr
    /var/db/freshports/ports-jail/usr/sbin
    /var/db/freshports/ports-jail/usr/bin
    /var/db/freshports/ports-jail/usr/share
    /var/db/freshports/ports-jail/usr/share/mk
    /var/db/freshports/ports-jail/usr/libexec
    /var/db/freshports/ports-jail/usr/lib
    /var/db/freshports/ports-jail/make-master-port-test.sh
    /var/db/freshports/ports-jail/vars.sh
    /var/db/freshports/ports-jail/libexec
    /var/db/freshports/ports-jail/etc
    /var/db/freshports/ports-jail/etc/make.conf
    /var/db/freshports/ports-jail/make-port.sh
    /var/db/freshports/ports-jail/var
    /var/db/freshports/ports-jail/var/db
    /var/db/freshports/ports-jail/var/db/repos
    /var/db/freshports/ports-jail/make-category-comment.sh
    /var/db/freshports/ports-jail/dev
    /var/db/freshports/ports-jail/realpath.sh
    /var/db/freshports/ports-jail/vars.sh.sample
    /var/db/freshports/ports-jail/make-showconfig.sh
    /var/db/freshports/.bash_history
    /var/db/freshports/mail
    /var/db/freshports/signals
    /var/db/freshports/message-queues
    /var/db/freshports/message-queues/archive
    /var/db/freshports/message-queues/retry
    /var/db/freshports/message-queues/recent
    /var/db/freshports/tmp
    /var/db/freshports/cache
    /var/db/freshports/cache/spooling
    /var/db/freshports/cache/html
    /var/db/freshports/cache/daily
    /var/db/freshports/cache/ports
    /var/db/freshports/cache/news