Aug 142021
 

This post is the latest in a series of posts documenting the process of converting from using a chroot to using a full proper jail.

I spent about 2 hours of this fine Saturday morning writing this up and carrying out the steps.

As a result, both dev and test are now using a FreeBSD jail to extract data from ports in order to populate the database.

These are working notes, not so much a tutorial. However, you might find it useful too. I know I will when it comes to to convert stage tomorrow.

When you see mkjail below, that’s from sysutils/mkjail.

In this post:

  • on the jail – means the parent jail
  • on the child jail – means the child jail
  • on the host – means the server, the main server, hosting all the jails
    1. These are the steps used to update test-ingress01 from using ~freshports/ports-jail to /jails/freshports
      2021-08-14
      
      
      on the host:
      
      sudo zfs snapshot system/jails/test-ingress01@before-moving-to-jail-freshports
      
      on the jail:
      
      sudo service freshports stop
      sudo service ingress stop
      
      sudo sysrc freshports_enable="NO"
      sudo sysrc ingress_enable="NO"
      
      
      on the host:
      
      sudo service jail stop test-ingress01
      
      sudo mv /etc/fstab.test-ingress01 /etc/fstab.test-ingress01.NOT-IN-USE
      
      
      sudo zfs create -o mountpoint=none nvd/freshports/jailed/test-ingress01
      sudo zfs create -o mountpoint=none nvd/freshports/jailed/test-ingress01/jails
      sudo zfs create -o mountpoint=none nvd/freshports/jailed/test-ingress01/mkjail
      
      
      sudoedit /etc/jail.conf
      
      Add this to test-ingress01 jail definition:
      
          allow.chflags;
      
          # added when trying to get devfs in subjails
          allow.mount.fdescfs;
      
          allow.mount;
          allow.mount.devfs;
          allow.mount.linprocfs;
          allow.mount.nullfs;
          allow.mount.procfs;
          allow.mount.tmpfs = 1;
          allow.mount.zfs;
      
          allow.raw_sockets;
          allow.socket_af;
      
          children.max=6;
      
          exec.created+="zfs set jailed=on nvd/freshports/jailed/test-ingress01";
          exec.created+="zfs jail $name    nvd/freshports/jailed/test-ingress01";
      
          exec.poststart  += "jail -m allow.mount.linprocfs=1 name=$name";
      
          enforce_statfs=1;
      
      EDIT: 2021-09-09 - enforce_statfs was missing from the above - re https://twitter.com/DLangille/status/1435965521959215109
      
      
      Start the jail:
      
      sudo service jail start test-ingress01
      
      
      in the jail:
      
      verify zfs
      
      [dan@test-ingress01:~] $ zfs list
      [dan@test-ingress01:~] $ zfs list
      NAME                                          USED  AVAIL  REFER  MOUNTPOINT
      nvd                                          81.7G   133G    23K  none
      nvd/freshports                               81.5G   133G    23K  none
      nvd/freshports/jailed                        2.48G   133G    24K  none
      nvd/freshports/jailed/test-ingress01           72K   133G    24K  none
      nvd/freshports/jailed/test-ingress01/jails     24K   133G    24K  none
      nvd/freshports/jailed/test-ingress01/mkjail    24K   133G    24K  none
      [dan@test-ingress01:~] $ 
      
      
      Set mount points:
      
      sudo zfs set mountpoint=/jails         nvd/freshports/jailed/test-ingress01/jails
      sudo zfs set mountpoint=/var/db/mkjail nvd/freshports/jailed/test-ingress01/mkjail
      
      
      [dan@test-ingress01:~] $ zfs list
      NAME                                          USED  AVAIL  REFER  MOUNTPOINT
      nvd                                          81.7G   133G    23K  none
      nvd/freshports                               81.5G   133G    23K  none
      nvd/freshports/jailed                        2.48G   133G    24K  none
      nvd/freshports/jailed/test-ingress01           72K   133G    24K  none
      nvd/freshports/jailed/test-ingress01/jails     24K   133G    24K  /jails
      nvd/freshports/jailed/test-ingress01/mkjail    24K   133G    24K  /var/db/mkjail
      [dan@test-ingress01:~] $ 
      
      
      
      Creating the jail within a jail:
      
      sudo pkg install mkjail
      
      sudoedit /usr/local/etc/mkjail.conf :
      
      ZPOOL="nvd"
      JAILDATASET="freshports/jailed/test-ingress01/jails"
      
      sudo mkjail create -a amd64 -j freshports -v 12.2-RELEASE
      
      If that fails with:
      
      tar: could not chdir to '/12.2-RELEASE/'
      
      Then run the same command again.
      
      sudo sysrc jail_enable="YES"
      
      su to root, enter a bash shell, and run this:
      
      cat << EOF > /etc/jail.conf
      exec.start = "/bin/sh /etc/rc";
      exec.stop = "/bin/sh /etc/rc.shutdown";
      exec.clean;
      mount.devfs;
      path = /jails/\$name;
      allow.raw_sockets;
      securelevel = 2;
      exec.consolelog="/var/tmp/jail-\$name";
      
      host.hostname = "\$name\$(hostname)";
      
      persist;
      freshports {
          host.hostname = "freshports";
      
          ip4 = inherit;
          persist;
      
          devfs_ruleset=0;
      
          allow.mount=true;
          enforce_statfs=1;
          allow.mount.devfs;
          allow.mount.procfs;
      }
      EOF
      
      mkdir -p /jails/freshports/usr/local/etc/pkg/repos
      echo "FreeBSD: { enabled: no }" > /jails/freshports/usr/local/etc/pkg/repos/FreeBSD.conf
      
      cat << EOF > /jails/freshports/usr/local/etc/pkg/repos/local.conf
      local: {
         url: "pkg+http://fedex.unixathome.org/packages/122amd64-default-master-list/"
         mirror_type: "srv",
         signature_type: "PUBKEY",
         pubkey: "/etc/ssl/slocum.unixathome.org.cert",   
         enabled: true
      }
      EOF
      
      
      cat << EOF > /jails/freshports/etc/ssl/slocum.unixathome.org.cert
      -----BEGIN PUBLIC KEY-----
      MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAuveHTXwrwGmdWG6oFWgN
      R/7bOQiQIE9iFmXmy4/MX9+01zMh2mjTfIOFe8IE1QqOK7X5EkVqhIdHCFg1Cg/x
      Gl5oxxQEhp/HQbtVN7pNcpXKIGl0t5sFcjwXQqXS1wb87JP+v6KuOSTCFzS3l/X2
      XtOo5QJZtxuF7+IM2PZlYb8MDmhVxriPw0pWRiP8lyW2LV1dSrN+VFpllZYinfIv
      Sr7sUArIVlHH+Ddrm5MWjqTsHid36og26NDmAQnfX96IBF1sXadSTxKy3YwCnEcv
      L7mBJhNyTIULuSbknM9zP9amkrlyLhWl+SdRGRkcOmXwHgzbdZsL62OlkXYIgkB+
      tK099ARziCSe+sclhgfjoixnXxk0h9gUU6h5BDafATvtP4KDmwDYQEXO/7OPS0/H
      vHFLEWLExcbW6hF0fyy2aA/HTlX83bBqJ+evsFtcvyxNfDp7tnjus/oAmJ82IU4F
      0ZmWMoJDeNznO+3iokHH0J8vxa4kd1hjoSaDh1+qOZCXGRcsqyDTgWQEdQ5Uy76j
      c9NgCJqHqJyjPqsiIZmPJNGUh1f7VaSUT8a131X5dwj6kx02s55UGJeIlK6F1d1f
      7+7pdfv4rzHK3iPlP1eXQlv8szGAxdDbkSFqLK6gIC8V/Mf0B0zN0aU1JZkaAvoH
      GopjN8IyF6fx/5yN9EAp8GUCAwEAAQ==
      -----END PUBLIC KEY-----
      EOF
      
      
      cat << EOF > /jails/freshports/etc/resolv.conf
      search unixathome.org int.unixathome.org
      nameserver 10.55.0.1 
      nameserver 10.55.0.73
      nameserver 10.55.0.13
      EOF
      
      
      cat << EOF > /jails/freshports/etc/rc.conf
      cron_enable="NO"
      syslogd_enable="NO"
      sendmail_enable="NO"
      sendmail_submit_enable="NO"
      sendmail_outbound_enable="NO"
      sendmail_msp_queue_enable="NO"
      EOF
      
      
      service jail start
      
      jexec freshports
      
      Nothing but this should be running:
      
      root@freshports:/ # ps auwwx
      USER  PID %CPU %MEM   VSZ  RSS TT  STAT STARTED    TIME COMMAND
      root 8106  0.0  0.0 13196 3800  5  SJ   14:11   0:00.01 /bin/csh -i
      root 8376  0.0  0.0 11768 2776  5  R+J  14:12   0:00.00 ps auwwx
      
      pkg install pkg
      
      exit
      
      Now, back in the parent jail
      
      cd /jails/freshports/usr
      sudo git clone https://git.FreeBSD.org/ports.git
      
      
      
      sudo mv ~freshports/ports-jail ~freshports/ports-jail.NO.LONGER.IN.USE
      
      sudo pkg upgrade
      
      cd /usr/local/etc/freshports
      
      reconcile the configuration files:
      
      sudoedit config.ini config.ini.sample
      
      sudoedit config.pm config.pm.sample
      sudo diff -ruN config.pm config.pm.sample | less
      
      sudoedit config.sh config.sh.sample
      sudo diff -ruN config.sh config.sh.sample  | less
      
      
      On ansible:
      
      ansible-playbook freshports-ingress-git.yml --limit=test-ingress01.int.unixathome.org --tags=sudoers
      
      
      on the parent jail:
      
      sudo sysrc ingress_enable="YES"
      sudo service ingress start
      
      Watch the logs, wait for some commits.
      
      sudo service ingress stop
      
      cd /var/db/ingress/message-queues/incoming
      sudo mv -i * ../holding/
      
      sudo sysrc freshports_enable="YES"
      sudo service freshports start
      cd ../holding/
      
      Move one of the xml files into incoming:
      
      sudo mv -i BLAH.xml ../incoming/
      
      check for errors, repeat until confidence is high.
      
      Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive