FreshPorts pulls down a lot of distfiles (the file distributed by the software author which contains the software). Much more than your average user. Every distfile for every commit is fetched. Let’s look at my private FreshPorts development instance:
[13:18 dvl-ingress01 dvl /jails/freshports/usr/ports] % du -ch -d 0 distfiles 125G distfiles 125G total
That’s 125GB. Sometimes, though, it gets a bit out of control. Like this node:
[13:22 r720-02-ingress01 dvl ~] % du -ch -d 0 /jails/freshports/usr/ports/distfiles 306G /jails/freshports/usr/ports/distfiles 306G total
What’s in there, let’s look at one part:
[13:23 r720-02-ingress01 dvl ~] % ls /jails/freshports/usr/ports/distfiles/z* /jails/freshports/usr/ports/distfiles/zabbix-4.0.44.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-4.0.45.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-4.0.46.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-4.0.47.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-4.0.48.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-4.0.49.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-4.0.50.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.30.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.31.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.32.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.33.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.34.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.35.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.36.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.37.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.38.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.39.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.40.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.41.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.42.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.43.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-5.0.44.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.12.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.13.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.14.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.15.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.16.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.17.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.18.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.19.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.20.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.21.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.22.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.23.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.24.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.25.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.26.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.27.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.28.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.29.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.30.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.31.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.32.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.33.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.34.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.0.35.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.2.6.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.2.7.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.2.8.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.2.9.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.0.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.1.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.10.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.11.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.12.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.13.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.14.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.15.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.16.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.17.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.18.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.19.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.2.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.3.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.4.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.5.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.6.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.7.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.8.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-6.4.9.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-7.0.0.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-7.0.1.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-7.0.2.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-7.0.3.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-7.0.4.tar.gz /jails/freshports/usr/ports/distfiles/zabbix-7.0.5.tar.gz /jails/freshports/usr/ports/distfiles/zalando-go-keyring-62750a1ff80d_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zchee-go-vmnet-97ebf9174097_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zclconf-go-cty-v1.1.1_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zdharma-continuum-fast-syntax-highlighting-v1.55_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zeek-5.0.4.tar.gz /jails/freshports/usr/ports/distfiles/zeek-5.0.5.tar.gz /jails/freshports/usr/ports/distfiles/zeek-5.0.6.tar.gz /jails/freshports/usr/ports/distfiles/zeek-5.0.7.tar.gz /jails/freshports/usr/ports/distfiles/zeek-5.0.8.tar.gz /jails/freshports/usr/ports/distfiles/zeek-5.0.9.tar.gz /jails/freshports/usr/ports/distfiles/zeek-6.0.0.tar.gz /jails/freshports/usr/ports/distfiles/zeek-6.0.1.tar.gz /jails/freshports/usr/ports/distfiles/zeek-6.0.2.tar.gz /jails/freshports/usr/ports/distfiles/zeek-6.0.3.tar.gz /jails/freshports/usr/ports/distfiles/zeek-6.0.4.tar.gz /jails/freshports/usr/ports/distfiles/zeek-7.0.0.tar.gz /jails/freshports/usr/ports/distfiles/zeek-7.0.1.tar.gz /jails/freshports/usr/ports/distfiles/zeek-7.0.2.tar.gz /jails/freshports/usr/ports/distfiles/zeek-7.0.3.tar.gz /jails/freshports/usr/ports/distfiles/zeek-zeek-netmap-v2.0.0_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zenphoto-zenphoto-v1.5.9_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zenphoto-zenphoto-v1.6.1_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zenphoto-zenphoto-v1.6.2_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zenphoto-zenphoto-v1.6.3_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zenphoto-zenphoto-v1.6.4_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zenphoto-zenphoto-v1.6.5_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zenphoto-zenphoto-v1.6_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zfs-snapshot-mgmt-20090201.tar.gz /jails/freshports/usr/ports/distfiles/zh-auto-cn-l10n-1.1.tar.gz /jails/freshports/usr/ports/distfiles/zip30.tar.gz /jails/freshports/usr/ports/distfiles/zmb3-spotify-v1.3.0_GH0.tar.gz /jails/freshports/usr/ports/distfiles/znc-1.8.2.tar.gz /jails/freshports/usr/ports/distfiles/znc-1.9.0.tar.gz /jails/freshports/usr/ports/distfiles/znc-1.9.1.tar.gz /jails/freshports/usr/ports/distfiles/zoneminder-zoneminder-1.36.12_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zoneminder-zoneminder-1.36.33_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zoneminder-zoneminder-1.36.34_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zoneminder-zoneminder-1.36.35_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zorkian-go-datadog-api-v2.29.0_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zserge-jsmn-18e9fe42cbfe21d65076f5c77ae2be3_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zsh-5.9-doc.tar.xz /jails/freshports/usr/ports/distfiles/zsh-5.9.tar.xz /jails/freshports/usr/ports/distfiles/zsh-users-antigen-v2.2.3_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zsh-users-zsh-autosuggestions-v0.7.0_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zsh-users-zsh-syntax-highlighting-0.7.1_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zsh-users-zsh-syntax-highlighting-0.8.0_GH0.tar.gz /jails/freshports/usr/ports/distfiles/zangband: zangband-2.7.5pre1.tar.gz
FreshPorts pulls down each distfile for each version. It never cleans up. This sacrifices disk space in exchange for less time spent downloading. However, the old distfiles are never used. Only the latest distfile is of interest. If, for some odd reason, we needed an older distfile, I’m willing to spend the time and bandwidth to pull it down again. When might the non-current distfile be needed? Perhaps for a commit on a branch; granted, that’s probably rare.
The argument against trimming
Disk space is cheap. Code is buggy. Why bother? The old distfiles are never used. Why keep them? I had to talk myself out of the ‘just leave them be’ attitude.
Eventually the disk will fill because of this. That’s not likely at home, but it will happen on virtual hosts, where disk space is often limited by GB, not TB.
The call for help
I posted on Mastodon saying I wanted to delete old distfiles. The best answer so far came from Petre who suggested using INDEX. The solution:
* Compile a list of the distfiles from INDEX
* delete everything not on that list.
INDEX can be fetched, or built. Building takes me about 13 minutes. Fetching takes about X seconds.
The code
The code is:
# cut -d \| -f 2 ${PORTSDIR}/INDEX* | while read d; do cd $d && make -V ALLFILES | tr " " "\n" | grep -v '^$' || : ; done > current-distfiles
# find ${DISTDIR} -type f | grep -v -f current-distfiles | xargs rm
Separate filesystem
First task: move the distfiles directory to a separate filesystem. This will allow me to test repeatedly and rollback and have the distfiles to delete again.
[dvl@r720-02:~] $ sudo zfs create data01/freshports/jailed/ingress01/distfiles [dvl@r720-02:~] $ sudo zfs snapshot data01/freshports/jailed/ingress01/distfiles@empty [dvl@r720-02:~] $ zfs list -r data01/freshports/jailed/ingress01 NAME USED AVAIL REFER MOUNTPOINT data01/freshports/jailed/ingress01 311G 1.17T 24K none data01/freshports/jailed/ingress01/distfiles 24K 1.17T 24K none data01/freshports/jailed/ingress01/jails 310G 1.17T 24K /jails data01/freshports/jailed/ingress01/jails/freshports 310G 1.17T 310G /jails/freshports data01/freshports/jailed/ingress01/mkjail 1.37G 1.17T 798M /var/db/mkjail data01/freshports/jailed/ingress01/mkjail/14.1-RELEASE 605M 1.17T 605M /var/db/mkjail/14.1-RELEASE [dvl@r720-02:~] $ sudo zfs jail ingress01 data01/freshports/jailed/ingress01/distfiles [dvl@r720-02:~] $
Now let’s swap from the host (above) into the jail on that host:
[13:42 r720-02-ingress01 dvl ~] % zfs list NAME USED AVAIL REFER MOUNTPOINT data01 599G 1.17T 23K /data01 data01/freshports 311G 1.17T 23K none data01/freshports/jailed 311G 1.17T 24K none data01/freshports/jailed/ingress01 311G 1.17T 24K none data01/freshports/jailed/ingress01/distfiles 24K 1.17T 24K none data01/freshports/jailed/ingress01/jails 310G 1.17T 24K /jails data01/freshports/jailed/ingress01/jails/freshports 310G 1.17T 310G /jails/freshports data01/freshports/jailed/ingress01/mkjail 1.37G 1.17T 798M /var/db/mkjail data01/freshports/jailed/ingress01/mkjail/14.1-RELEASE 605M 1.17T 605M /var/db/mkjail/14.1-RELEASE [13:42 r720-02-ingress01 dvl ~] % sudo zfs set mountpoint=/jails/freshports/usr/ports/distfiles.new data01/freshports/jailed/ingress01/distfiles [13:43 r720-02-ingress01 dvl ~] % zfs list NAME USED AVAIL REFER MOUNTPOINT data01 599G 1.17T 23K /data01 data01/freshports 311G 1.17T 23K none data01/freshports/jailed 311G 1.17T 24K none data01/freshports/jailed/ingress01 311G 1.17T 24K none data01/freshports/jailed/ingress01/distfiles 24K 1.17T 24K /jails/freshports/usr/ports/distfiles.new data01/freshports/jailed/ingress01/jails 310G 1.17T 24K /jails data01/freshports/jailed/ingress01/jails/freshports 310G 1.17T 310G /jails/freshports data01/freshports/jailed/ingress01/mkjail 1.37G 1.17T 798M /var/db/mkjail data01/freshports/jailed/ingress01/mkjail/14.1-RELEASE 605M 1.17T 605M /var/db/mkjail/14.1-RELEASE [13:44 r720-02-ingress01 dvl ~] % cd /jails/freshports/usr/ports [13:44 r720-02-ingress01 dvl /jails/freshports/usr/ports] % ls -ld distfiles* drwxr-xr-x 138 root wheel 7157 2024.11.04 19:42 distfiles/ drwxr-xr-x 2 root wheel 2 2024.11.05 13:40 distfiles.new/
Now, let’s move it:
[13:45 r720-02-ingress01 dvl /jails/freshports/usr/ports] % sudo mv distfiles/* distfiles.new
That took about 35 minutes.
Next, I moved away the existing, and moved in the new:
[14:19 r720-02-ingress01 dvl /jails/freshports/usr/ports] % ls -ld distfiles* drwxr-xr-x 2 root wheel 2 2024.11.05 14:19 distfiles/ drwxr-xr-x 138 root wheel 7157 2024.11.05 14:19 distfiles.new/ [14:20 r720-02-ingress01 dvl /jails/freshports/usr/ports] % sudo mv distfiles distfiles.old [14:20 r720-02-ingress01 dvl /jails/freshports/usr/ports] % sudo zfs set mountpoint=/jails/freshports/usr/ports/distfiles data01/freshports/jailed/ingress01/distfiles [14:20 r720-02-ingress01 dvl /jails/freshports/usr/ports] % ls -ld distfiles* drwxr-xr-x 138 root wheel 7157 2024.11.05 14:19 distfiles/ drwxr-xr-x 2 root wheel 2 2024.11.05 13:43 distfiles.new/ drwxr-xr-x 2 root wheel 2 2024.11.05 14:19 distfiles.old/ [14:20 r720-02-ingress01 dvl /jails/freshports/usr/ports] % sudo zfs snapshot data01/freshports/jailed/ingress01/distfiles@all-distfiles
Lastly, I took a snapshot of the new contents. This will let me redo any experiments with the new script.
Testing the script
The first draft of the file is at this GitHub gist.
I ran it through these two filters:
grep -v 'not found' gistfile1.txt > gistfile2.txt grep -v 'returned non-zero status' gistfile2.txt > gistfile3.txt grep -v 'Unable to determine compiler type' gistfile3.txt > gistfile4.txt grep -v " " gistfile4.txt > gistile5.txt
In hindsight, perhaps just the last filter would be sufficient.
Now let’s try the delete, or the example delete, and see if we can get an idea of what’s going to be deleted.
Adjusting the path
Oh, wait, we have a path problem:
[18:13 r720-02-ingress01 dvl ~/tmp] % find /jails/freshports/usr/ports/distfiles/ -type f | head /jails/freshports/usr/ports/distfiles/clamav-1.4.0.tar.gz /jails/freshports/usr/ports/distfiles/msva-nginx_ajp_module-fcbb2cc_GH0.tar.gz /jails/freshports/usr/ports/distfiles/redis-6.0.20.tar.gz /jails/freshports/usr/ports/distfiles/mitchellh-mapstructure-v1.1.2_GH0.tar.gz /jails/freshports/usr/ports/distfiles/apcupsd-3.14.14.tar.gz /jails/freshports/usr/ports/distfiles/afni-afni-AFNI_23.2.02_GH0.tar.gz /jails/freshports/usr/ports/distfiles/mitchellh-go-testing-interface-v1.0.0_GH0.tar.gz /jails/freshports/usr/ports/distfiles/mitre-fieldmanual-cb53f2b_GH0.tar.gz /jails/freshports/usr/ports/distfiles/OSLV-Monitor-0.1.0.tar.gz /jails/freshports/usr/ports/distfiles/DtxdF-AppJail-vg20231102-11b5191b324e22ec59c614dc42dbae24ec6c9822_GH0.tar.gz
Let’s try this instead:
[18:17 r720-02-ingress01 dvl /jails/freshports/usr/ports/distfiles] % find . -type f | grep -v -f ~/tmp/gistfile5.txt grep: trailing backslash (\)
What’s that all about?
% grep '\\' ~/tmp/gistfile5.txt \i3lock-2.15.tar.xz
I don’t know why that is there. Perhaps I typed it.
Trying again, still the path problem
[18:20 r720-02-ingress01 dvl /jails/freshports/usr/ports/distfiles] % find . -type f | grep -v -f ~/tmp/gistfile5.txt [18:20 r720-02-ingress01 dvl /jails/freshports/usr/ports/distfiles] % find . -type f | head ./clamav-1.4.0.tar.gz ./msva-nginx_ajp_module-fcbb2cc_GH0.tar.gz ./redis-6.0.20.tar.gz ./mitchellh-mapstructure-v1.1.2_GH0.tar.gz ./apcupsd-3.14.14.tar.gz ./afni-afni-AFNI_23.2.02_GH0.tar.gz ./mitchellh-go-testing-interface-v1.0.0_GH0.tar.gz ./mitre-fieldmanual-cb53f2b_GH0.tar.gz ./OSLV-Monitor-0.1.0.tar.gz ./DtxdF-AppJail-vg20231102-11b5191b324e22ec59c614dc42dbae24ec6c9822_GH0.tar.gz
2024-12-24
It’s time to do this on the production host.
Let’s look at before:
root@freshports:/ # find /usr/ports/distfiles/ | wc -l
4675
root@freshports:/ # du -ch -d 0 /usr/ports/distfiles/
79G /usr/ports/distfiles/
79G total
First, fetch the INDEX:
root@freshports:/usr/ports # make fetch index
This is the script:
root@freshports:/ # cat list-current-distfiles.sh
#!/bin/sh
PORTSDIR=/usr/ports
cut -d \| -f 2 ${PORTSDIR}/INDEX* | while read d; do cd $d && make -V ALLFILES | tr " " "\n" | grep -v '^$' || : ; done
Run like this:
root@freshports:/ # sh list-old-distfiles.sh > /tmp/distfiles-to-keep ld-elf.so.1: /lib/libcxxrt.so.1: version CXXABI_1.3.11 required by /lib/libc++.so.1 not found make: "/usr/ports/Mk/Uses/compiler.mk" line 105: warning: "/usr/bin/clang --version" returned non-zero status make: "/usr/ports/Mk/Uses/compiler.mk" line 134: warning: "c++ -### /dev/null 2>&1" returned non-zero status .... make: "/usr/ports/Mk/Uses/compiler.mk" line 105: warning: "/usr/bin/clang --version" returned non-zero status make: "/usr/ports/Mk/Uses/compiler.mk" line 134: warning: "c++ -### /dev/null 2>&1" returned non-zero status root@freshports:/ # less /tmp/distfiles-to-delete accerciser-3.38.0.tar.xz at-spi2-core-2.54.0.tar.xz atkmm-2.28.4.tar.xz caribou-0.4.21.tar.xz adrg-xdg-v0.3.3_GH0.tar.gz ...
We have this many files to keep, if we have them already:
root@freshports:/ # wc -l /tmp/distfiles-to-keep 182133 /tmp/distfiles-to-keep
This command takes the list of files in the directory, ignores anything current, and removes the rest:
root@freshports:/ # find /usr/ports/distfiles -type f | grep -v -f /tmp/current-distfiles | xargs rm
The results
Now we have only 33G used and have deleted 1001 files.
root@freshports:/ # find /usr/ports/distfiles/ | wc -l
3674
root@freshports:/ # du -ch -d 0 /usr/ports/distfiles/
33G /usr/ports/distfiles/
33G total
