Today, FreshPorts broke. No commits were being processed. nullfs mounts were disappearing. No idea why.
Let’s follow the clues.
All times are local to me (Philadelphia).
12:46 pm
Rene contacted me on IRC, presumably about a failed sanity test I had seen earlier in the day:
To: rene@FreeBSD.org From: FreshPorts Sanity Daemon <FreshPorts@FreshPorts.org> Cc: dan@langille.org Subject: FreshPorts sanity checking Date: Fri, 10 Aug 2018 16:39:24 +0000 Message-ID: <20180810_163924_037160.FreshPorts@FreshPorts.org> This message was generated by the FreshPorts Daemon. You recently made this commit, which FreshPorts FreshPorts had trouble processing: MessageID: 201808101638.w7AGcsUL078691@repo.freebsd.org Subject : svn commit: r476841 - head/www/feedjack The following is a list of the ports which had errors: http://www.freshports.org/www/feedjack/ The exact errors appear below. www/feedjack: This command (FreshPorts code 1): /usr/local/bin/sudo /usr/sbin/chroot -u freshports /var/db/freshports/ports-jail /make-port.sh /var/db/repos/PORTS-head www/feedjack 2>/tmp/FreshPorts.www.feedjack.make-error.2018.8.10.16.39.24.59977 produced this error: Error message is: chroot: /make-port.sh: No such file or directory You are receiving this message as you are a FreeBSD committer who has opted into this service. Please see https://www.FreshPorts.org/committer-opt-in.php for more information. -- FreshPorts Daemon
You can look at Rene’s commit.
Looking back, Rene’s was not the first commit to encounter this issue, but I didn’t notice the other failures. It wasn’t until I got a “FAILED: MASTER_PORT” notice that I knew something was wrong. MASTER_PORT is a routine test to exercise the code which determines the MASTER_PORT value. If it gets the wrong answer for sysutils/bacula-server, I know something is broken.
Now let’s examine the command mentioned in the sanity test failure email:
/usr/local/bin/sudo /usr/sbin/chroot -u freshports /var/db/freshports/ports-jail \
/make-port.sh /var/db/repos/PORTS-head www/feedjack
A chroot is invoked on the /var/db/freshports/ports-jail directory as the freshports user. The command /make-port.sh /var/db/repos/PORTS-head www/feedjack is run.
If you want to read more about ports-jail and it’s script, try this search.
I tried running the command manually:
$ /usr/local/bin/sudo /usr/sbin/chroot -u freshports /var/db/freshports/ports-jail chroot: /bin/csh: No such file or directory
I thought: Why does it care about /bin/csh? I’m not running that, nor do I need it.
What’s in the jail?
$ ls -l /var/db/freshports/ports-jail total 50 drwxr-xr-x 2 root freshports 2 Oct 30 2017 bin -r-xr-xr-x 1 root wheel 617 Aug 3 04:31 cat-descr.sh drwxr-xr-x 2 root freshports 2 Oct 30 2017 dev drwxr-xr-x 2 root wheel 3 Aug 7 13:57 etc drwxr-xr-x 2 root freshports 2 Oct 30 2017 lib drwxr-xr-x 2 root freshports 2 Oct 30 2017 libexec -r-xr-xr-x 1 root wheel 593 Aug 3 04:31 make-category-comment.sh -r-xr-xr-x 1 root wheel 707 Aug 3 04:31 make-generate-plist.sh -r-xr-xr-x 1 root wheel 640 Aug 3 04:31 make-master-port-test.sh -r-xr-xr-x 1 root wheel 627 Aug 3 04:31 make-master-sites-all.sh -r-xr-xr-x 1 root wheel 1536 Aug 3 04:31 make-port.sh -r-xr-xr-x 1 root wheel 606 Aug 3 04:31 make-showconfig.sh -r-xr-xr-x 1 root wheel 601 Aug 3 04:31 realpath.sh drwxr-xr-x 2 root freshports 2 Oct 30 2017 sbin drwxr-xr-x 7 root wheel 7 Oct 30 2017 usr drwxr-xr-x 3 freshports freshports 3 Nov 1 2017 var drwxr-xr-x 3 root wheel 3 Oct 6 2017 var.original -r-xr-xr-x 1 root wheel 196 Aug 3 04:31 vars.sh -r-xr-xr-x 1 root wheel 196 Aug 3 04:31 vars.sh.sample
OK, make-port.sh is there. What about the bin directory?
$ ls -l /var/db/freshports/ports-jail/bin total 0
Oh. That should not be empty. This is the problem.
iocage
The FreshPorts production server uses iocage for jail management. The extra nullfs mounts are lists below and are explained in this blog post.
[dan@x8dtu:~] $ sudo iocage fstab -l x8dtu-ingress01 +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | INDEX | FSTAB ENTRY | +=======+===============================================================================================================================================================================+ | 0 | | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | /iocage/jails/x8dtu-ingress01/root/bin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/bin nullfs ro,nosuid 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 2 | none /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/dev devfs rw 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 3 | /iocage/jails/x8dtu-ingress01/root/lib /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/lib nullfs ro,nosuid 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 4 | /iocage/jails/x8dtu-ingress01/root/libexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/libexec nullfs ro,nosuid 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 5 | /iocage/jails/x8dtu-ingress01/root/sbin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/sbin nullfs ro,nosuid 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 6 | /iocage/jails/x8dtu-ingress01/root/usr/share/mk /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/share/mk nullfs ro,nosuid,noexec 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 7 | /iocage/jails/x8dtu-ingress01/root/usr/sbin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/sbin nullfs ro,nosuid 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 8 | /iocage/jails/x8dtu-ingress01/root/usr/bin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/bin nullfs ro,nosuid 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 9 | /iocage/jails/x8dtu-ingress01/root/usr/lib /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/lib nullfs ro,nosuid 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 10 | /iocage/jails/x8dtu-ingress01/root/usr/libexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/libexec nullfs ro,nosuid 0 0 | +-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [dan@x8dtu:~] $
The above is what should be mounted. Compare to below:
[dan@x8dtu:~] $ mount | grep x8dtu-ingress01 main_tank/iocage/jails/x8dtu-ingress01 on /iocage/jails/x8dtu-ingress01 (zfs, local, noatime, nfsv4acls) main_tank/iocage/jails/x8dtu-ingress01/root on /iocage/jails/x8dtu-ingress01/root (zfs, local, noatime, nfsv4acls) zroot/data/ingress01-testing on /iocage/jails/x8dtu-ingress01/root/usr/home/dan/tmp-fast (zfs, local, noatime, nfsv4acls) main_tank/data/freshports/backend/cache on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/cache (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/backend/cache/html on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/cache/html (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/backend/cache/spooling on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/cache/spooling (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/backend/queues/archive on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/message-queues/archive (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/backend/queues/recent on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/message-queues/recent (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/backend/queues/retry on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/message-queues/retry (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail (zfs, local, noatime, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var/db on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var/db/repos on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos (zfs, local, noatime, noexec, nosuid, nfsv4acls) zroot/data/freshports/repo/PORTS-head on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos/PORTS-head (zfs, local, noatime, nfsv4acls) main_tank/data/freshports/backend/queues/incoming on /iocage/jails/x8dtu-ingress01/root/var/db/ingress/message-queues/incoming (zfs, local, noatime, noexec, nosuid, nfsv4acls) devfs on /iocage/jails/x8dtu-ingress01/root/dev (devfs, local, multilabel) fdescfs on /iocage/jails/x8dtu-ingress01/root/dev/fd (fdescfs)
Yeah, that’s hard to read.
How about this instead:
[dan@x8dtu:~] $ mount | grep /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail main_tank/data/freshports/ports-jail on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail (zfs, local, noatime, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var/db on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var/db/repos on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos (zfs, local, noatime, noexec, nosuid, nfsv4acls) zroot/data/freshports/repo/PORTS-head on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos/PORTS-head (zfs, local, noatime, nfsv4acls) [dan@x8dtu:~] $
Yeah, stuff is missing. Let’s just restart the jail
[dan@x8dtu:~] $ sudo iocage restart x8dtu-ingress01 * Stopping x8dtu-ingress01 + Running prestop OK + Stopping services OK + Removing jail process OK + Running poststop OK * Starting x8dtu-ingress01 + Started OK + Starting services OK
Now we see them all:
[dan@x8dtu:~] $ mount | grep /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail main_tank/data/freshports/ports-jail on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail (zfs, local, noatime, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var/db on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var/db/repos on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos (zfs, local, noatime, noexec, nosuid, nfsv4acls) zroot/data/freshports/repo/PORTS-head on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos/PORTS-head (zfs, local, noatime, nfsv4acls) /iocage/jails/x8dtu-ingress01/root/bin on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/bin (nullfs, local, nosuid, read-only) devfs on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/dev (devfs, local, multilabel) /iocage/jails/x8dtu-ingress01/root/lib on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/lib (nullfs, local, nosuid, read-only) /iocage/jails/x8dtu-ingress01/root/libexec on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/libexec (nullfs, local, nosuid, read-only) /iocage/jails/x8dtu-ingress01/root/sbin on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/sbin (nullfs, local, nosuid, read-only) /iocage/jails/x8dtu-ingress01/root/usr/share/mk on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/share/mk (nullfs, local, noexec, nosuid, read-only) /iocage/jails/x8dtu-ingress01/root/usr/sbin on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/sbin (nullfs, local, nosuid, read-only) /iocage/jails/x8dtu-ingress01/root/usr/bin on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/bin (nullfs, local, nosuid, read-only) /iocage/jails/x8dtu-ingress01/root/usr/lib on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/lib (nullfs, local, nosuid, read-only) /iocage/jails/x8dtu-ingress01/root/usr/libexec on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/libexec (nullfs, local, nosuid, read-only) [dan@x8dtu:~] $
I went to check on the jail. It was empty
[dan@x8dtu:~] $ ls /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/bin [dan@x8dtu:~] $
What? It’s mounted.
No, it’s not mounted:
[dan@x8dtu:/var/log] $ mount | grep /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail main_tank/data/freshports/ports-jail on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail (zfs, local, noatime, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var/db on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db (zfs, local, noatime, noexec, nosuid, nfsv4acls) main_tank/data/freshports/ports-jail/var/db/repos on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos (zfs, local, noatime, noexec, nosuid, nfsv4acls) zroot/data/freshports/repo/PORTS-head on /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/var/db/repos/PORTS-head (zfs, local, noatime, nfsv4acls) [dan@x8dtu:/var/log] $
In the short time between restarting the jail and checking the directory, the nullfs mounts were gone.
What is doing this? Nothing in logs, nothing to be found anywhere.
The story so far
- The FreshPorts server consists of several jails, one of which uses a chroot, which I call a ports-jail, to differentiate it from a FreeBSD jail.
- The ports-jail uses 10 nullfs mounts.
- The nullfs mounts are present when the jail starts.
- The nullfs mounts disappear sometime later.
- What is removing these mounts? There was nothing in any logs.
This whole situation had me very confused.
2:20 pm
I amended /etc/fstab to contain all the mounts required by the x8dtu-ingress01 jail. This allowed me to easily test the issue by getting everything remounted easily via mount -a.
I kept monitoring the number of mountpoints via mount | wc -l. 82 was wrong. 92 was right.
I was looking at my script and trying to figure out what step was causing the issue. It didn’t happen every time. I was playing around with sleep 5 and other variations, trying to figure out if it was a timing issue.
I didn’t get anywhere with that.
2:45 pm
By this time, I was convinced something odd was going on with the server, so I rebooted it. Nothing was making sense.
2:59 pm
I open a ticket with the ISP because I can’t get to the FreshPorts server and my IPMI link is not working.
I was thinking I had message up the bootcode when I recently upgraded the zpool. I thought I had missed doing:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot …
3:31 pm
After a few short discussions, my ISP showed me this screen shot of the FreshPorts server.
OK, so that’s my fault. My edits at 2:20 pm to /etc/fstab have come back to haunt me.
3:41 pm
IPMI access has been restored. I remove my modification from /etc/fstab and reboot the server.
3:45 pm
The server is back online, but FreshPorts is still not healthy. The nullfs mounts are still disappearing and I have no idea why.
4:03 pm
A suggestion on IRC arrives:
[Aug 10 16:03] <@markj> dvl: if you have root access, you can try running
"dtrace -n 'fbt::nullfs_unmount:entry {printf("%s (ppid %d)", curpsinfo->pr_psargs, curpsinfo->pr_ppid);}'"
to get some idea of where the unmount is coming from
I run the command. A short time later:
[dan@x8dtu:~] $ sudo dtrace -n 'fbt::nullfs_unmount:entry {printf("%s (ppid %d)", curpsinfo->pr_psargs, curpsinfo->pr_ppid);}'
dtrace: description 'fbt::nullfs_unmount:entry ' matched 1 probe
CPU ID FUNCTION:NAME
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 23475)
13 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-nginx01/fstab (ppid 23492)
13 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-nginx01/fstab (ppid 23492)
While markj and I are discussing this, more entries appear:
6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 24858) 6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 24858) 6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 24858) 6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 24858) 6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 24858) 6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 24858) 6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 24858) 6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 24858) 6 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-ingress01/fstab (ppid 24858) 15 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-nginx01/fstab (ppid 24882) 15 66394 nullfs_unmount:entry umount -a -F /iocage/jails/x8dtu-nginx01/fstab (ppid 24882)
I am really suspecting iocage now. The fstab files mentioned above are present on disk:
[dan@x8dtu:/var/log] $ ls -l /iocage/jails/*/fstab -rw-r--r-- 1 root wheel 0 Oct 3 2017 /iocage/jails/mx-ingress01/fstab -rw-r--r-- 1 root wheel 1768 Aug 10 18:20 /iocage/jails/x8dtu-ingress01/fstab -rw-r--r-- 1 root wheel 368 Oct 29 2017 /iocage/jails/x8dtu-nginx01/fstab -rw-r--r-- 1 root wheel 0 Sep 9 2017 /iocage/jails/x8dtu-pg01/fstab -rw-r--r-- 1 root wheel 0 Aug 7 21:03 /iocage/jails/x8dtu-pg02.vpn.unixathome.org/fstab
I can see they are used by both x8dtu-ingress01 and x8dtu-nginx01. The file contents is what you saw for iocage fstab -l earlier in this post.
NOTE: pid 23475 was no longer around, but knowing what that was would have helped us.
4:14pm
markj has another idea:
[Aug 10 16:11] <@markj> oh, dwatch can get the process tree. try "dwatch -R fbt::nullfs_mount:entry'" instead [Aug 10 16:14] <@markj> dvl: the dwatch invocation should show more info about the ancestors of the umount process. hopefully enough to figure out what's going on
A short while later:
[dan@x8dtu:~] $ sudo dwatch -R fbt::nullfs_mount:entry
INFO Watching 'fbt::nullfs_mount:entry' ...
2018 Aug 10 20:16:48 0.0 mount_nullfs[26186]: mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/bin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/bin
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26175 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
\-+= 26185 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
\-+= 26186 0.0 mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/bin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/bin
2018 Aug 10 20:16:48 0.0 mount_nullfs[26187]: mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/lib /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/lib
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26175 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
\-+= 26185 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
\-+= 26187 0.0 mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/lib /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/lib
2018 Aug 10 20:16:48 0.0 mount_nullfs[26188]: mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/libexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/libexec
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26175 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
\-+= 26185 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
\-+= 26188 0.0 mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/libexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/libexec
2018 Aug 10 20:16:48 0.0 mount_nullfs[26189]: mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/sbin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/sbin
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26175 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
\-+= 26185 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
\-+= 26189 0.0 mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/sbin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/sbin
2018 Aug 10 20:16:48 0.0 mount_nullfs[26190]: mount_nullfs -o ro -o nosuid -o noexec /iocage/jails/x8dtu-ingress01/root/usr/share/mk /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/share/mk
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26175 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
\-+= 26185 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
\-+= 26190 0.0 mount_nullfs -o ro -o nosuid -o noexec /iocage/jails/x8dtu-ingress01/root/usr/share/mk /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/share/mk
2018 Aug 10 20:16:48 0.0 mount_nullfs[26191]: mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/usr/sbin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/sbin
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26175 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
\-+= 26185 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
\-+= 26191 0.0 mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/usr/sbin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/sbin
2018 Aug 10 20:16:48 0.0 mount_nullfs[26192]: mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/usr/bin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/bin
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26175 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
\-+= 26185 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
\-+= 26192 0.0 mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/usr/bin /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/bin
2018 Aug 10 20:16:48 0.0 mount_nullfs[26193]: mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/usr/lib /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/lib
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26175 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
\-+= 26185 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
\-+= 26193 0.0 mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/usr/lib /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/lib
2018 Aug 10 20:16:48 0.0 mount_nullfs[26194]: mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/usr/libexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/libexec
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26175 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-ingress01 file /bin/sh
\-+= 26185 0.0 mount -a -F /iocage/jails/x8dtu-ingress01/fstab
\-+= 26194 0.0 mount_nullfs -o ro -o nosuid /iocage/jails/x8dtu-ingress01/root/usr/libexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/ports-jail/usr/libexec
2018 Aug 10 20:16:49 0.0 mount_nullfs[26209]: mount_nullfs -o ro -o nosuid -o noexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/cache/html /iocage/jails/x8dtu-nginx01/root/var/db/freshports/cache/html
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26198 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-nginx01 file /bin/sh
\-+= 26208 0.0 mount -a -F /iocage/jails/x8dtu-nginx01/fstab
\-+= 26209 0.0 mount_nullfs -o ro -o nosuid -o noexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/cache/html /iocage/jails/x8dtu-nginx01/root/var/db/freshports/cache/html
2018 Aug 10 20:16:49 0.0 mount_nullfs[26210]: mount_nullfs -o ro -o nosuid -o noexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/signals /iocage/jails/x8dtu-nginx01/root/var/db/freshports/signals
-+= 26131 0.0 /bin/sh /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
\-+= 26198 0.0 /usr/local/bin/python3.6 /usr/local/bin/iocage chroot x8dtu-nginx01 file /bin/sh
\-+= 26208 0.0 mount -a -F /iocage/jails/x8dtu-nginx01/fstab
\-+= 26210 0.0 mount_nullfs -o ro -o nosuid -o noexec /iocage/jails/x8dtu-ingress01/root/var/db/freshports/signals /iocage/jails/x8dtu-nginx01/root/var/db/freshports/signals
Ahah! It’s one of my Nagios check scripts, /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh
The script
This was the script in use. Do not use it. You’ll find out why later.
[dan@x8dtu:/usr/local/libexec/nagios-custom] $ cat check_py_iocage_host_vs_jails.sh
#!/bin/sh
HOSTVERSION=`/usr/bin/file /bin/sh`
JAILS=`/usr/local/bin/iocage list -H | /usr/bin/cut -f 2 -w`
ERRORS=''
for jail in ${JAILS}
do
JAILVERSION=`/usr/local/bin/iocage chroot ${jail} 'file /bin/sh'`
if [ "${JAILVERSION}" != "${HOSTVERSION}" ]
then
ERRORS="jail '${jail}' is ${JAILVERSION}"
fi
done
if [ "${ERRORS}" == "" ]
then
echo 'All jails match the host'
exit 0
else
echo "WARNING: HOST and jails are NOT in sync: host = ${HOSTVERSION} but ${ERRORS}"
exit 2
fi
[dan@x8dtu:/usr/local/libexec/nagios-cuFalstom] $
This script is used to verify that the jail and the host are running the same version of world.
It was the iocage chroot command which was running umount.
I tried to track down why this was suddenly happening.
What changed?
I know Ansible was run earlier in the day. It was installing new scripts on each host for nrpe.
What happened? check_py_iocage_host_vs_jails.sh got updated.
My conclusion: local changes to the file on the x8dtu-ingress01 never pushed upstream.
Find the original
I checked ZFS snapshots. None.
I checked Bacula. I didn’t backup /usr/local/libexec/nagios-custom, presumably because it is in Ansible, but I do now.
Fail. No original script found.
4:30 pm Stop the bleeding
Now that I knew the cause, I commented out that entry in /usr/local/etc/nrpe.cfg and restarted nrpe3.
I restarted the two jails to fix their mount points.
The messed up commits, three or four of them, were rerun.
Success. The website is caught up and online.
Tweet sent.
Status page updated.
Original script found
I found the original script. Ansible backed it up, as instructed.
It was at /usr/local/libexec/nagios-custom/check_py_iocage_host_vs_jails.sh.39690.2018-08-10@15:59:42~
What was the difference?
# svn di
Index: check_py_iocage_host_vs_jails.sh
===================================================================
--- check_py_iocage_host_vs_jails.sh (revision 1197)
+++ check_py_iocage_host_vs_jails.sh (working copy)
@@ -2,11 +2,11 @@
HOSTVERSION=`/usr/bin/file /bin/sh`
-JAILS=`/usr/local/bin/iocage list -H | /usr/bin/cut -f 2 -w`
+JAILS=`/usr/local/bin/iocage list | /usr/bin/egrep -v '^\| JID' | grep '^|' | /usr/bin/cut -f 4 -w`
ERRORS=''
for jail in ${JAILS}
do
- JAILVERSION=`/usr/local/bin/iocage chroot ${jail} 'file /bin/sh'`
+ JAILVERSION=`/usr/local/bin/iocage exec ${jail} 'file /bin/sh'`
if [ "${JAILVERSION}" != "${HOSTVERSION}" ]
then
ERRORS="jail '${jail}' is ${JAILVERSION}"
I’ll keep the line 10 and ignore line 11. Line 10 is better.
Line 16 is the faulty one. exec is better than chroot.
What does the man page say?
chroot Chroot into a jail without actually starting the jail
itself. Useful for initial setup like setting a root
password or configuring networking. A command can be
specified as with the normal system, see chroot(8).
exec Execute a command inside the specified jail. This is an
iocage UUID/NAME wrapper for jexec(8). After invoking
exec, specify the jail, any commands to run inside that
jail, and any arguments for those commands.
To me, they should both work, but they do not. Clearly, chroot was mounting and umounting the fstab contents, by design.
I prefer chroot because it means the script can test non-running jails, but obviously I had to use exec instead.
Notes
- I committed that ‘new’ script into the repo used by Ansible
- Don’t work local, work on Ansible.
- If you do work local, commit it back.
- Backup your important files, even if they are supposed to be in Ansible
- When things went wrong, I first suspected something Ansible did. I commented out the two new commands added to nrpe.cfg.
- When I suspected Ansible, I should have looked around the /usr/local/libexec/nagios-custom directory and I’d have noticed the updated files.
- I should learn more about dtrace and dwatch, especially on how to form a simple query.
- Perhaps I should be monitoring mount points. At least then I’d have had more clues.
6:49 pm
My thanks to markj who provided the dtrace and dwatch queries which helped track down my mistake.
It’s pub time. Good night.
