Recently, a new use of ZFS snapshots was introduced for a cache on FreshPorts. This approach involves creating a snapshot on the empty cache, then rolling back to that snapshot when the cache needs to be cleared.
This idea came from this Tweet after much discussion on how to properly and safely delete directory contents without hitting a race condition. The zfs rollback is much neater.
This approach is now in place on all FreshPorts servers.
The original FreshPorts server was in a small tower chassis. The first deployed-to-a-datacenter server was a dual-Opteron with 8 x 74GB Raptor drives. This host was deployed in 2006 and is still in use. It remained the primary FreshPorts server until November 2017 when it was replaced by a new system from iXsystems.
More about the Opteron server.
This server is not running ZFS and hardware RAID. Converting to ZFS remotely is not going to happen.
I have been putting off upgrading this server to the latest FreshPorts code because of this.
Last night I realized I could deploy that code by creating a ZFS file system within a regular UFS file.
The ZFS file
From man zpool, don’t do this at home:
file A regular file. The use of files as a backing store is strongly discouraged. It is designed primarily for experimental purposes, as the fault tolerance of a file is only as good the file system of which it is a part. A file must be specified by a full path.
For this specific purpose, I am accepting the risk.
First step, enable and start ZFS.
[dan@supernews:~] $ sudo sysrc zfs_enable="YES" [dan@supernews:~] $ sudo service zfs start [dan@supernews:~] $ zfs list no datasets available
When I went looking for how to create the file, I found Ubuntu 16.04 – Using Files To Test ZFS and the truncate command was used.
How big should this file be? Let’s look at production.
[dan@x8dtu-nginx01:~] $ zfs list main_tank/data/freshports/backend/cache/packages NAME USED AVAIL REFER MOUNTPOINT main_tank/data/freshports/backend/cache/packages 7.90M 3.47T 7.84M /var/db/freshports/cache/packages [dan@x8dtu-nginx01:~] $ sudo du -ch -d0 /var/db/freshports/cache/packages 8.3M /var/db/freshports/cache/packages 8.3M total
Let’s say that is 9MB.
How many files are in there?
[dan@x8dtu-nginx01:~] $ sudo find /var/db/freshports/cache/packages | wc -l 1356
There are packages for 1356 ports cached. Let’s say there are 40,000 packages. That would be 9MB / 1356 * 40,000 = 265 MB
Let’s call it 1GB for good measure. This is only for caching.
After consulting man truncate, I settled on this command:
[dan@supernews:~/tmp] $ truncate -s 1G packages [dan@supernews:~/tmp] $ ls -l packages -rw-r--r-- 1 dan dan 1073741824 Jun 28 12:59 packages [dan@supernews:~/tmp] $
Before I start messing with the website cache, let’s turn off the services which use it:
[dan@supernews:/var/db/freshports/cache] $ sudo service nginx stop Stopping nginx. Waiting for PIDS: 78270. [dan@supernews:/var/db/freshports/cache] $ sudo svc -d /var/service/freshports [dan@supernews:/var/db/freshports/cache] $ sudo svc -d /var/service/fp-listen [dan@supernews:/var/db/freshports/cache] $
The two svc items are the commit processing daemon and the cache clearing daemon, respectively.
This moves the file into place:
[dan@supernews:/var/db/freshports] $ sudo mv ~/tmp/packages freshports-zfs [dan@supernews:/var/db/freshports] $ sudo chown root:wheel freshports-zfs
Creating the zpool
This creates the zpool:
$ sudo zpool create freshports /var/db/freshports/freshports-zfs $ zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT freshports 960M 87.5K 960M - - 0% 0% 1.00x ONLINE -
It works, just like any other zpool:
[dan@supernews:/var/db/freshports/cache] $ zpool status pool: freshports state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM freshports ONLINE 0 0 0 /var/db/freshports/freshports-zfs ONLINE 0 0 0 errors: No known data errors [dan@supernews:/var/db/freshports/cache] $ sudo zpool scrub freshports [dan@supernews:/var/db/freshports/cache] $ zpool status pool: freshports state: ONLINE scan: scrub repaired 0 in 0 days 00:00:00 with 0 errors on Sun Jun 28 14:29:26 2020 config: NAME STATE READ WRITE CKSUM freshports ONLINE 0 0 0 /var/db/freshports/freshports-zfs ONLINE 0 0 0 errors: No known data errors [dan@supernews:/var/db/freshports/cache] $
Next up, this zpool needs a filesystem.
Creating the filesystem
I don’t need atime, we don’t need compression here, and let’s set the mountpoint.
Compression is usually a good thing. I’m deliberately choosing not to use it. I can’t justify. It just feels wrong to do this on top of UFS. This isn’t anything about UFS. It’s just me.
$ sudo zfs create -o atime=off -o compression=off -o mountpoint=/var/db/freshports/cache/packages freshports/packages $ zfs list NAME USED AVAIL REFER MOUNTPOINT freshports 175K 832M 23K /freshports freshports/packages 23K 832M 23K /var/db/freshports/cache/packages
Oh wait, I don’t like having /freshports mounted.
$ sudo zfs set mountpoint=none canmount=off freshports $ zfs list NAME USED AVAIL REFER MOUNTPOINT freshports 188K 832M 23K none freshports/packages 23K 832M 23K /var/db/freshports/cache/packages
What do the permissions look like?
$ ls -l /var/db/freshports/cache total 27619 drwxrwxr-- 27 www freshports 512 Apr 20 08:19 categories drwxrwxr-x 3 www freshports 512 Jan 23 14:22 daily drwxr-xr-x 2 www freshports 512 Apr 20 08:21 general drwxr-xr-x 2 freshports freshports 512 Jun 28 14:33 html drwxrwxr-x 2 www freshports 512 Jun 28 06:10 news drwxr-xr-x 2 root wheel 2 Jun 28 14:34 packages drwxrwxr-- 14 www freshports 512 Jun 23 12:00 packages.old drwxr-xr-x 2 www freshports 512 Nov 9 2018 pages drwxrwxr-x 41 www freshports 2048 May 18 22:42 ports -rw-r--r-- 1 www www 28234059 Jun 6 05:42 searchlog.txt drwxrwxr-x 2 www freshports 512 Jun 28 14:33 spooling [dan@supernews:/var/db/freshports/cache] $
That’s wrong, fixing it:
$ sudo chown www:freshports packages
I must snapshot now, before making any data changes which I do not want to rollback.
$ sudo zfs snapshot freshports/packages@empty
Missing from this post, as I discovered last night, delegation.
Right now, it does not work:
$ echo zfs rollback freshports/packages@empty | sudo su -fm freshports cannot rollback 'freshports/packages': permission denied
I must allow the freshports user to do the rollback.
sudo zfs allow freshports rollback freshports/packages
Now it does:
$ echo zfs rollback freshports/packages@empty | sudo su -fm freshports $
It only took me two months and a failed FreshPorts server to find out.
Moving over the data
Now I move over the existing cache:
$ sudo mv -i packages.old/ packages/ $ cd packages/ $ ls packages.old
Or… not. Damn.
$ sudo mv packages.old/* $ sudo rmdir packages.old/ $ ls comms devel java mail multimedia www databases emulators lang misc net-p2p x11-toolkits
Back into service
Let’s start up everything I stopped.
$ sudo svc -u /var/service/freshports $ sudo svc -u /var/service/fp-listen $ sudo service nginx start Performing sanity check on nginx configuration: nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful Starting nginx.
Next, this server needs to have the latest code installed. That’s not for today.