By “post correction”, I meant “post-correction script”, i.e. after the correction script completed.
A recent commit started the process of moving the WWW from pkg-descr to Makefile. These were huge commits, about 27930 ports each. FreshPorts is dumb. It sees a commit and it refreshes all ports touched by that commit. Refreshing every port takes a long time. It’s not just running make -V and extracting values. It looks at DEPENDS, pkg-plist, refreshes secondary ports based on changes to primary ports. Some ports take a second to do this. Some take 5-7 seconds. Even at 1 second per port, we’re looking at about 8 hours to process these commits.
That’s not the big issue. The bigger issue was the clearing out of all existing homepage values. With no WWW entry in pkg-descr, FreshPorts set that value as emtpy in the database.
The solution: update the code to use make -V WWW and if empty, take the value from pkg-descr. This “if empty” option would allow quarterly branch ports to continue to work.
The diff for the code change is not big.
With new code packaged and a refresh-ports scripts ready to go, I started rolling the changes out to dev, test, stage, and prod. This whole process took about about a week, based on this first tweet.
The results
[/sourcecode]
The purpose of this blog post is to record any differences between the various FreshPorts databases in case that become relevant later.
The goal: how many [active] ports have empty homepages? By active, I mean non-deleted ports; ports you would find in a git clone output.
dev
On dev, we have 1465 such ports.
freshports.devgit=# SELECT count(*)
FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
AND (PA.homepage is null or trim(PA.homepage) = '');
count
-------
1465
(1 row)
[sourcecode]
<h3>test</h3>
On <span class="cmd">test</span>, we have 1465 such ports.
[sourcecode]
freshports.testgit=# SELECT count(*)
FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
AND (PA.homepage is null or trim(PA.homepage) = '');
count
-------
1465
(1 row)
[sourcecode]
<h3>stage</h3>
On <span class="cmd">stage</span>, we have 1347 such ports.
[sourcecode]
You are now connected to database "freshports.stagegit" as user "dan".
freshports.stagegit=# SELECT count(*)
FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
AND (PA.homepage is null or trim(PA.homepage) = '');
count
-------
1347
(1 row)
[sourcecode]
<h3>prod</h3>
On <span class="cmd">prod</span>, we have 1465 such ports.
[sourcecode]
freshports.devgit=# SELECT count(*)
FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
AND (PA.homepage is null or trim(PA.homepage) = '');
count
-------
1465
(1 row)
[sourcecode]
<h2>What's next?</h2>
I want to see the differences between these results. Which ports are different? That shouldn't take much. A short script to pull out the results. A few diff statements.
<h3>dev vs stage</h3>
The results have been captured into files:
<pre class="screen">
[pro02 dan ~/tmp/homepages] % wc -l *
1464 dev
1345 prod
1346 stage
1464 test
5619 total
</pre>
It looks like <span class="file">dev</span> and <span class="file">test</span> match:
[sourcecode]
[pro02 dan ~/tmp/homepages] % diff -ruN dev test
[pro02 dan ~/tmp/homepages] %
The big diff is dev vs stage and seem related to libreoffice:
[pro02 dan ~/tmp/homepages] % diff -ruN dev stage --- dev 2022-09-14 07:45:03.000000000 -0400 +++ stage 2022-09-14 07:43:47.000000000 -0400 @@ -2,7 +2,6 @@ /ports/head/arabic/ae_fonts_mono /ports/head/arabic/ae_fonts_ttf /ports/head/arabic/kacst_fonts - /ports/head/arabic/libreoffice /ports/head/archivers/deb2targz /ports/head/archivers/ha /ports/head/archivers/lcab @@ -87,8 +86,6 @@ /ports/head/chinese/hc /ports/head/chinese/jis2gb /ports/head/chinese/kcfonts - /ports/head/chinese/libreoffice-zh_CN - /ports/head/chinese/libreoffice-zh_TW /ports/head/chinese/lunar /ports/head/chinese/moettf /ports/head/chinese/ttfm @@ -312,109 +309,6 @@ /ports/head/editors/edith /ports/head/editors/hexpert /ports/head/editors/jove - /ports/head/editors/libreoffice-af - /ports/head/editors/libreoffice-am - /ports/head/editors/libreoffice-as - /ports/head/editors/libreoffice-ast - /ports/head/editors/libreoffice-be - /ports/head/editors/libreoffice-bg - /ports/head/editors/libreoffice-bn - /ports/head/editors/libreoffice-bn_IN - /ports/head/editors/libreoffice-bo - /ports/head/editors/libreoffice-br - /ports/head/editors/libreoffice-brx - /ports/head/editors/libreoffice-bs - /ports/head/editors/libreoffice-ca - /ports/head/editors/libreoffice-ca_valencia - /ports/head/editors/libreoffice-ckb - /ports/head/editors/libreoffice-cs - /ports/head/editors/libreoffice-cy - /ports/head/editors/libreoffice-da - /ports/head/editors/libreoffice-dgo - /ports/head/editors/libreoffice-dsb - /ports/head/editors/libreoffice-dz - /ports/head/editors/libreoffice-el - /ports/head/editors/libreoffice-en_GB - /ports/head/editors/libreoffice-en_ZA - /ports/head/editors/libreoffice-eo - /ports/head/editors/libreoffice-es - /ports/head/editors/libreoffice-et - /ports/head/editors/libreoffice-eu - /ports/head/editors/libreoffice-fa - /ports/head/editors/libreoffice-fi - /ports/head/editors/libreoffice-fur - /ports/head/editors/libreoffice-fy - /ports/head/editors/libreoffice-ga - /ports/head/editors/libreoffice-gd - /ports/head/editors/libreoffice-gl - /ports/head/editors/libreoffice-gu - /ports/head/editors/libreoffice-gug - /ports/head/editors/libreoffice-hi - /ports/head/editors/libreoffice-hr - /ports/head/editors/libreoffice-hsb - /ports/head/editors/libreoffice-id - /ports/head/editors/libreoffice-is - /ports/head/editors/libreoffice-it - /ports/head/editors/libreoffice-ka - /ports/head/editors/libreoffice-kab - /ports/head/editors/libreoffice-kk - /ports/head/editors/libreoffice-km - /ports/head/editors/libreoffice-kmr_Latn - /ports/head/editors/libreoffice-kn - /ports/head/editors/libreoffice-kok - /ports/head/editors/libreoffice-ks - /ports/head/editors/libreoffice-lb - /ports/head/editors/libreoffice-lo - /ports/head/editors/libreoffice-lt - /ports/head/editors/libreoffice-lv - /ports/head/editors/libreoffice-mai - /ports/head/editors/libreoffice-mk - /ports/head/editors/libreoffice-ml - /ports/head/editors/libreoffice-mn - /ports/head/editors/libreoffice-mni - /ports/head/editors/libreoffice-mr - /ports/head/editors/libreoffice-my - /ports/head/editors/libreoffice-nb - /ports/head/editors/libreoffice-ne - /ports/head/editors/libreoffice-nl - /ports/head/editors/libreoffice-nn - /ports/head/editors/libreoffice-nr - /ports/head/editors/libreoffice-nso - /ports/head/editors/libreoffice-oc - /ports/head/editors/libreoffice-om - /ports/head/editors/libreoffice-or - /ports/head/editors/libreoffice-pa_IN - /ports/head/editors/libreoffice-ro - /ports/head/editors/libreoffice-rw - /ports/head/editors/libreoffice-sa_IN - /ports/head/editors/libreoffice-sat - /ports/head/editors/libreoffice-sd - /ports/head/editors/libreoffice-si - /ports/head/editors/libreoffice-sid - /ports/head/editors/libreoffice-sk - /ports/head/editors/libreoffice-sl - /ports/head/editors/libreoffice-sq - /ports/head/editors/libreoffice-sr - /ports/head/editors/libreoffice-sr_Latn - /ports/head/editors/libreoffice-ss - /ports/head/editors/libreoffice-st - /ports/head/editors/libreoffice-sv - /ports/head/editors/libreoffice-sw_TZ - /ports/head/editors/libreoffice-szl - /ports/head/editors/libreoffice-ta - /ports/head/editors/libreoffice-te - /ports/head/editors/libreoffice-tg - /ports/head/editors/libreoffice-th - /ports/head/editors/libreoffice-tn - /ports/head/editors/libreoffice-tr - /ports/head/editors/libreoffice-ts - /ports/head/editors/libreoffice-tt - /ports/head/editors/libreoffice-ug - /ports/head/editors/libreoffice-uz - /ports/head/editors/libreoffice-ve - /ports/head/editors/libreoffice-vec - /ports/head/editors/libreoffice-xh - /ports/head/editors/libreoffice-zu /ports/head/editors/nvi-m17n /ports/head/editors/sted /ports/head/editors/uemacs @@ -429,7 +323,6 @@ /ports/head/emulators/vxtools /ports/head/finance/p5-Business-OnlinePayment-2CheckOut /ports/head/finance/p5-Business-OnlinePayment-PaymentsGateway - /ports/head/french/libreoffice /ports/head/ftp/fmirror /ports/head/ftp/ftpfind /ports/head/ftp/ftpmirror @@ -555,7 +448,6 @@ /ports/head/german/aspell /ports/head/german/eric6 /ports/head/german/hyphen - /ports/head/german/libreoffice /ports/head/german/phone /ports/head/graphics/4va /ports/head/graphics/bmp2html @@ -593,9 +485,7 @@ /ports/head/graphics/xoris /ports/head/graphics/xtexcad /ports/head/graphics/xwpick - /ports/head/hebrew/libreoffice /ports/head/hungarian/hyphen - /ports/head/hungarian/libreoffice /ports/head/hungarian/mythes /ports/head/irc/bnc /ports/head/irc/irc @@ -627,7 +517,6 @@ /ports/head/japanese/kinput2 /ports/head/japanese/kon2-14dot /ports/head/japanese/kon2-16dot - /ports/head/japanese/libreoffice /ports/head/japanese/libslang /ports/head/japanese/man /ports/head/japanese/man-doc @@ -655,7 +544,6 @@ /ports/head/korean/hmconv /ports/head/korean/hpscat /ports/head/korean/kaistfonts - /ports/head/korean/libreoffice /ports/head/korean/munhwafonts-cid /ports/head/korean/nanum-barun-gothic /ports/head/korean/nanum-barun-pen @@ -935,7 +823,6 @@ /ports/head/news/trn /ports/head/polish/fortunepl /ports/head/polish/hyphen - /ports/head/polish/libreoffice /ports/head/ports-mgmt/genpatch /ports/head/ports-mgmt/lsknobs /ports/head/ports-mgmt/pchecker @@ -954,8 +841,6 @@ /ports/head/ports-mgmt/wanted-ports /ports/head/portuguese/aspell-pt_PT /ports/head/portuguese/hyphen - /ports/head/portuguese/libreoffice - /ports/head/portuguese/libreoffice-pt_BR /ports/head/print/afm /ports/head/print/asprint /ports/head/print/bibcursed @@ -1002,7 +887,6 @@ /ports/head/russian/hyphen /ports/head/russian/koi2koi /ports/head/russian/koi8r-ps - /ports/head/russian/libreoffice /ports/head/russian/mythes /ports/head/russian/p5-XML-Parser-encodings /ports/head/security/ADMsmb @@ -1227,8 +1111,6 @@ /ports/head/textproc/xslide.el /ports/head/ukrainian/emacs-koi8u /ports/head/ukrainian/hyphen - /ports/head/ukrainian/libreoffice - /ports/head/vietnamese/libreoffice /ports/head/vietnamese/libviet /ports/head/vietnamese/unicode-uhoai /ports/head/vietnamese/urwvn [pro02 dan ~/tmp/homepages] % 7:46:13
Whereas, stage and prod have only one difference:
[pro02 dan ~/tmp/homepages] % diff -ruN stage prod --- stage 2022-09-14 07:43:47.000000000 -0400 +++ prod 2022-09-14 07:48:59.000000000 -0400 @@ -560,7 +560,6 @@ /ports/head/lang/emacs-lisp-intro /ports/head/lang/gcc-ecj45 /ports/head/lang/libobjc2 - /ports/head/lang/linux-dotnet11-sdk /ports/head/lang/pbasic /ports/head/lang/python-tools /ports/head/lang/ratfor [pro02 dan ~/tmp/homepages] %
It is time for more investigation.
