ZFS when you’re not doing ZFS

Recently, a new use of ZFS snapshots was introduced for a cache on FreshPorts. This approach involves creating a snapshot on the empty cache, then rolling back to that snapshot when the cache needs to be cleared.

This idea came from this Tweet after much discussion on how to properly and safely delete directory contents without hitting a race condition. The zfs rollback is much neater.

This approach is now in place on all FreshPorts servers.

Except one

The original FreshPorts server was in a small tower chassis. The first deployed-to-a-datacenter server was a dual-Opteron with 8 x 74GB Raptor drives. This host was deployed in 2006 and is still in use. It remained the primary FreshPorts server until November 2017 when it was replaced by a new system from iXsystems.

More about the Opteron server.

This server is not running ZFS and hardware RAID. Converting to ZFS remotely is not going to happen.

I have been putting off upgrading this server to the latest FreshPorts code because of this.

Last night I realized I could deploy that code by creating a ZFS file system within a regular UFS file.

The ZFS file

From man zpool, don’t do this at home:

     file    A regular file. The use of	files as a backing store is strongly
             discouraged. It is	designed primarily for experimental purposes,
             as	the fault tolerance of a file is only as good the file system
             of	which it is a part. A file must	be specified by	a full path.

For this specific purpose, I am accepting the risk.

First step, enable and start ZFS.

[dan@supernews:~] $ sudo sysrc zfs_enable="YES"
[dan@supernews:~] $ sudo service zfs start
[dan@supernews:~] $ zfs list
no datasets available

When I went looking for how to create the file, I found Ubuntu 16.04 – Using Files To Test ZFS and the truncate command was used.

How big should this file be? Let’s look at production.

[dan@x8dtu-nginx01:~] $ zfs list main_tank/data/freshports/backend/cache/packages
NAME                                               USED  AVAIL  REFER  MOUNTPOINT
main_tank/data/freshports/backend/cache/packages  7.90M  3.47T  7.84M  /var/db/freshports/cache/packages

[dan@x8dtu-nginx01:~] $ sudo du -ch -d0  /var/db/freshports/cache/packages
8.3M	/var/db/freshports/cache/packages
8.3M	total

Let’s say that is 9MB.

How many files are in there?

[dan@x8dtu-nginx01:~] $ sudo find /var/db/freshports/cache/packages | wc -l
    1356

There are packages for 1356 ports cached. Let’s say there are 40,000 packages. That would be 9MB / 1356 * 40,000 = 265 MB

Let’s call it 1GB for good measure. This is only for caching.

After consulting man truncate, I settled on this command:

[dan@supernews:~/tmp] $ truncate -s 1G packages
[dan@supernews:~/tmp] $ ls -l packages
-rw-r--r--  1 dan  dan  1073741824 Jun 28 12:59 packages
[dan@supernews:~/tmp] $

Before I start messing with the website cache, let’s turn off the services which use it:

[dan@supernews:/var/db/freshports/cache] $ sudo service nginx stop
Stopping nginx.
Waiting for PIDS: 78270.

[dan@supernews:/var/db/freshports/cache] $ sudo svc -d /var/service/freshports
[dan@supernews:/var/db/freshports/cache] $ sudo svc -d /var/service/fp-listen
[dan@supernews:/var/db/freshports/cache] $

The two svc items are the commit processing daemon and the cache clearing daemon, respectively.

This moves the file into place:

[dan@supernews:/var/db/freshports] $ sudo mv ~/tmp/packages freshports-zfs
[dan@supernews:/var/db/freshports] $ sudo chown root:wheel freshports-zfs

Creating the zpool

This creates the zpool:

$ sudo zpool create freshports /var/db/freshports/freshports-zfs
$ zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freshports   960M  87.5K   960M        -         -     0%     0%  1.00x  ONLINE  -

It works, just like any other zpool:

[dan@supernews:/var/db/freshports/cache] $ zpool status
  pool: freshports
 state: ONLINE
  scan: none requested
config:

	NAME                                 STATE     READ WRITE CKSUM
	freshports                           ONLINE       0     0     0
	  /var/db/freshports/freshports-zfs  ONLINE       0     0     0

errors: No known data errors
[dan@supernews:/var/db/freshports/cache] $ sudo zpool scrub freshports
[dan@supernews:/var/db/freshports/cache] $ zpool status
  pool: freshports
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:00 with 0 errors on Sun Jun 28 14:29:26 2020
config:

	NAME                                 STATE     READ WRITE CKSUM
	freshports                           ONLINE       0     0     0
	  /var/db/freshports/freshports-zfs  ONLINE       0     0     0

errors: No known data errors
[dan@supernews:/var/db/freshports/cache] $

Next up, this zpool needs a filesystem.

Creating the filesystem

I don’t need atime, we don’t need compression here, and let’s set the mountpoint.

Compression is usually a good thing. I’m deliberately choosing not to use it. I can’t justify. It just feels wrong to do this on top of UFS. This isn’t anything about UFS. It’s just me.

$ sudo zfs create -o atime=off -o compression=off -o mountpoint=/var/db/freshports/cache/packages freshports/packages

$ zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
freshports            175K   832M    23K  /freshports
freshports/packages    23K   832M    23K  /var/db/freshports/cache/packages

Oh wait, I don’t like having /freshports mounted.

$ sudo zfs set mountpoint=none canmount=off freshports

$ zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
freshports            188K   832M    23K  none
freshports/packages    23K   832M    23K  /var/db/freshports/cache/packages

What do the permissions look like?

$ ls -l /var/db/freshports/cache
total 27619
drwxrwxr--  27 www         freshports       512 Apr 20 08:19 categories
drwxrwxr-x   3 www         freshports       512 Jan 23 14:22 daily
drwxr-xr-x   2 www         freshports       512 Apr 20 08:21 general
drwxr-xr-x   2 freshports  freshports       512 Jun 28 14:33 html
drwxrwxr-x   2 www         freshports       512 Jun 28 06:10 news
drwxr-xr-x   2 root        wheel              2 Jun 28 14:34 packages
drwxrwxr--  14 www         freshports       512 Jun 23 12:00 packages.old
drwxr-xr-x   2 www         freshports       512 Nov  9  2018 pages
drwxrwxr-x  41 www         freshports      2048 May 18 22:42 ports
-rw-r--r--   1 www         www         28234059 Jun  6 05:42 searchlog.txt
drwxrwxr-x   2 www         freshports       512 Jun 28 14:33 spooling
[dan@supernews:/var/db/freshports/cache] $

That’s wrong, fixing it:

$ sudo chown www:freshports packages

Snapshot now!

I must snapshot now, before making any data changes which I do not want to rollback.

$ sudo zfs snapshot freshports/packages@empty

Edit 2020-08-25

Missing from this post, as I discovered last night, delegation.

Right now, it does not work:

$ echo zfs rollback freshports/packages@empty | sudo su -fm freshports
cannot rollback 'freshports/packages': permission denied

I must allow the freshports user to do the rollback.

sudo zfs allow freshports rollback freshports/packages

Now it does:

$ echo zfs rollback freshports/packages@empty | sudo su -fm freshports
$

It only took me two months and a failed FreshPorts server to find out.

Moving over the data

Now I move over the existing cache:

$ sudo mv -i packages.old/ packages/
$ cd packages/
$ ls
packages.old

Or… not. Damn.

$ sudo mv packages.old/* 
$ sudo rmdir packages.old/
$ ls
comms        devel        java         mail         multimedia   www
databases    emulators    lang         misc         net-p2p      x11-toolkits

Back into service

Let’s start up everything I stopped.

$ sudo svc -u /var/service/freshports
$ sudo svc -u /var/service/fp-listen
$ sudo service nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.