Sep 142022
 

By “post correction”, I meant “post-correction script”, i.e. after the correction script completed.

A recent commit started the process of moving the WWW from pkg-descr to Makefile. These were huge commits, about 27930 ports each. FreshPorts is dumb. It sees a commit and it refreshes all ports touched by that commit. Refreshing every port takes a long time. It’s not just running make -V and extracting values. It looks at DEPENDS, pkg-plist, refreshes secondary ports based on changes to primary ports. Some ports take a second to do this. Some take 5-7 seconds. Even at 1 second per port, we’re looking at about 8 hours to process these commits.

That’s not the big issue. The bigger issue was the clearing out of all existing homepage values. With no WWW entry in pkg-descr, FreshPorts set that value as emtpy in the database.

The solution: update the code to use make -V WWW and if empty, take the value from pkg-descr. This “if empty” option would allow quarterly branch ports to continue to work.

The diff for the code change is not big.

With new code packaged and a refresh-ports scripts ready to go, I started rolling the changes out to dev, test, stage, and prod. This whole process took about about a week, based on this first tweet.

The results

[/sourcecode]

The purpose of this blog post is to record any differences between the various FreshPorts databases in case that become relevant later.

The goal: how many [active] ports have empty homepages? By active, I mean non-deleted ports; ports you would find in a git clone output.

dev

On dev, we have 1465 such ports.

freshports.devgit=#   SELECT count(*)
    FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
  AND (PA.homepage is null or trim(PA.homepage) = '');
 count 
-------
  1465
(1 row)


<h3>test</h3>

On <span class="cmd">test</span>, we have 1465 such ports.


freshports.testgit=#   SELECT count(*)
    FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
  AND (PA.homepage is null or trim(PA.homepage) = '');
 count 
-------
  1465
(1 row)


<h3>stage</h3>

On <span class="cmd">stage</span>, we have 1347 such ports.


You are now connected to database "freshports.stagegit" as user "dan".
freshports.stagegit=#   SELECT count(*)
    FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
  AND (PA.homepage is null or trim(PA.homepage) = '');
 count 
-------
  1347
(1 row)


<h3>prod</h3>

On <span class="cmd">prod</span>, we have 1465 such ports.


freshports.devgit=#   SELECT count(*)
    FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
  AND (PA.homepage is null or trim(PA.homepage) = '');
 count 
-------
  1465
(1 row)


<h2>What's next?</h2>

I want to see the differences between these results. Which ports are different? That shouldn't take much. A short script to pull out the results. A few diff statements.

<h3>dev vs stage</h3>

The results have been captured into files:

<pre class="screen">
[pro02 dan ~/tmp/homepages] % wc -l *
    1464 dev
    1345 prod
    1346 stage
    1464 test
    5619 total
</pre>

It looks like <span class="file">dev</span> and <span class="file">test</span> match:


[pro02 dan ~/tmp/homepages] % diff -ruN dev test
[pro02 dan ~/tmp/homepages] % 

The big diff is dev vs stage and seem related to libreoffice:

[pro02 dan ~/tmp/homepages] % diff -ruN dev stage
--- dev	2022-09-14 07:45:03.000000000 -0400
+++ stage	2022-09-14 07:43:47.000000000 -0400
@@ -2,7 +2,6 @@
  /ports/head/arabic/ae_fonts_mono
  /ports/head/arabic/ae_fonts_ttf
  /ports/head/arabic/kacst_fonts
- /ports/head/arabic/libreoffice
  /ports/head/archivers/deb2targz
  /ports/head/archivers/ha
  /ports/head/archivers/lcab
@@ -87,8 +86,6 @@
  /ports/head/chinese/hc
  /ports/head/chinese/jis2gb
  /ports/head/chinese/kcfonts
- /ports/head/chinese/libreoffice-zh_CN
- /ports/head/chinese/libreoffice-zh_TW
  /ports/head/chinese/lunar
  /ports/head/chinese/moettf
  /ports/head/chinese/ttfm
@@ -312,109 +309,6 @@
  /ports/head/editors/edith
  /ports/head/editors/hexpert
  /ports/head/editors/jove
- /ports/head/editors/libreoffice-af
- /ports/head/editors/libreoffice-am
- /ports/head/editors/libreoffice-as
- /ports/head/editors/libreoffice-ast
- /ports/head/editors/libreoffice-be
- /ports/head/editors/libreoffice-bg
- /ports/head/editors/libreoffice-bn
- /ports/head/editors/libreoffice-bn_IN
- /ports/head/editors/libreoffice-bo
- /ports/head/editors/libreoffice-br
- /ports/head/editors/libreoffice-brx
- /ports/head/editors/libreoffice-bs
- /ports/head/editors/libreoffice-ca
- /ports/head/editors/libreoffice-ca_valencia
- /ports/head/editors/libreoffice-ckb
- /ports/head/editors/libreoffice-cs
- /ports/head/editors/libreoffice-cy
- /ports/head/editors/libreoffice-da
- /ports/head/editors/libreoffice-dgo
- /ports/head/editors/libreoffice-dsb
- /ports/head/editors/libreoffice-dz
- /ports/head/editors/libreoffice-el
- /ports/head/editors/libreoffice-en_GB
- /ports/head/editors/libreoffice-en_ZA
- /ports/head/editors/libreoffice-eo
- /ports/head/editors/libreoffice-es
- /ports/head/editors/libreoffice-et
- /ports/head/editors/libreoffice-eu
- /ports/head/editors/libreoffice-fa
- /ports/head/editors/libreoffice-fi
- /ports/head/editors/libreoffice-fur
- /ports/head/editors/libreoffice-fy
- /ports/head/editors/libreoffice-ga
- /ports/head/editors/libreoffice-gd
- /ports/head/editors/libreoffice-gl
- /ports/head/editors/libreoffice-gu
- /ports/head/editors/libreoffice-gug
- /ports/head/editors/libreoffice-hi
- /ports/head/editors/libreoffice-hr
- /ports/head/editors/libreoffice-hsb
- /ports/head/editors/libreoffice-id
- /ports/head/editors/libreoffice-is
- /ports/head/editors/libreoffice-it
- /ports/head/editors/libreoffice-ka
- /ports/head/editors/libreoffice-kab
- /ports/head/editors/libreoffice-kk
- /ports/head/editors/libreoffice-km
- /ports/head/editors/libreoffice-kmr_Latn
- /ports/head/editors/libreoffice-kn
- /ports/head/editors/libreoffice-kok
- /ports/head/editors/libreoffice-ks
- /ports/head/editors/libreoffice-lb
- /ports/head/editors/libreoffice-lo
- /ports/head/editors/libreoffice-lt
- /ports/head/editors/libreoffice-lv
- /ports/head/editors/libreoffice-mai
- /ports/head/editors/libreoffice-mk
- /ports/head/editors/libreoffice-ml
- /ports/head/editors/libreoffice-mn
- /ports/head/editors/libreoffice-mni
- /ports/head/editors/libreoffice-mr
- /ports/head/editors/libreoffice-my
- /ports/head/editors/libreoffice-nb
- /ports/head/editors/libreoffice-ne
- /ports/head/editors/libreoffice-nl
- /ports/head/editors/libreoffice-nn
- /ports/head/editors/libreoffice-nr
- /ports/head/editors/libreoffice-nso
- /ports/head/editors/libreoffice-oc
- /ports/head/editors/libreoffice-om
- /ports/head/editors/libreoffice-or
- /ports/head/editors/libreoffice-pa_IN
- /ports/head/editors/libreoffice-ro
- /ports/head/editors/libreoffice-rw
- /ports/head/editors/libreoffice-sa_IN
- /ports/head/editors/libreoffice-sat
- /ports/head/editors/libreoffice-sd
- /ports/head/editors/libreoffice-si
- /ports/head/editors/libreoffice-sid
- /ports/head/editors/libreoffice-sk
- /ports/head/editors/libreoffice-sl
- /ports/head/editors/libreoffice-sq
- /ports/head/editors/libreoffice-sr
- /ports/head/editors/libreoffice-sr_Latn
- /ports/head/editors/libreoffice-ss
- /ports/head/editors/libreoffice-st
- /ports/head/editors/libreoffice-sv
- /ports/head/editors/libreoffice-sw_TZ
- /ports/head/editors/libreoffice-szl
- /ports/head/editors/libreoffice-ta
- /ports/head/editors/libreoffice-te
- /ports/head/editors/libreoffice-tg
- /ports/head/editors/libreoffice-th
- /ports/head/editors/libreoffice-tn
- /ports/head/editors/libreoffice-tr
- /ports/head/editors/libreoffice-ts
- /ports/head/editors/libreoffice-tt
- /ports/head/editors/libreoffice-ug
- /ports/head/editors/libreoffice-uz
- /ports/head/editors/libreoffice-ve
- /ports/head/editors/libreoffice-vec
- /ports/head/editors/libreoffice-xh
- /ports/head/editors/libreoffice-zu
  /ports/head/editors/nvi-m17n
  /ports/head/editors/sted
  /ports/head/editors/uemacs
@@ -429,7 +323,6 @@
  /ports/head/emulators/vxtools
  /ports/head/finance/p5-Business-OnlinePayment-2CheckOut
  /ports/head/finance/p5-Business-OnlinePayment-PaymentsGateway
- /ports/head/french/libreoffice
  /ports/head/ftp/fmirror
  /ports/head/ftp/ftpfind
  /ports/head/ftp/ftpmirror
@@ -555,7 +448,6 @@
  /ports/head/german/aspell
  /ports/head/german/eric6
  /ports/head/german/hyphen
- /ports/head/german/libreoffice
  /ports/head/german/phone
  /ports/head/graphics/4va
  /ports/head/graphics/bmp2html
@@ -593,9 +485,7 @@
  /ports/head/graphics/xoris
  /ports/head/graphics/xtexcad
  /ports/head/graphics/xwpick
- /ports/head/hebrew/libreoffice
  /ports/head/hungarian/hyphen
- /ports/head/hungarian/libreoffice
  /ports/head/hungarian/mythes
  /ports/head/irc/bnc
  /ports/head/irc/irc
@@ -627,7 +517,6 @@
  /ports/head/japanese/kinput2
  /ports/head/japanese/kon2-14dot
  /ports/head/japanese/kon2-16dot
- /ports/head/japanese/libreoffice
  /ports/head/japanese/libslang
  /ports/head/japanese/man
  /ports/head/japanese/man-doc
@@ -655,7 +544,6 @@
  /ports/head/korean/hmconv
  /ports/head/korean/hpscat
  /ports/head/korean/kaistfonts
- /ports/head/korean/libreoffice
  /ports/head/korean/munhwafonts-cid
  /ports/head/korean/nanum-barun-gothic
  /ports/head/korean/nanum-barun-pen
@@ -935,7 +823,6 @@
  /ports/head/news/trn
  /ports/head/polish/fortunepl
  /ports/head/polish/hyphen
- /ports/head/polish/libreoffice
  /ports/head/ports-mgmt/genpatch
  /ports/head/ports-mgmt/lsknobs
  /ports/head/ports-mgmt/pchecker
@@ -954,8 +841,6 @@
  /ports/head/ports-mgmt/wanted-ports
  /ports/head/portuguese/aspell-pt_PT
  /ports/head/portuguese/hyphen
- /ports/head/portuguese/libreoffice
- /ports/head/portuguese/libreoffice-pt_BR
  /ports/head/print/afm
  /ports/head/print/asprint
  /ports/head/print/bibcursed
@@ -1002,7 +887,6 @@
  /ports/head/russian/hyphen
  /ports/head/russian/koi2koi
  /ports/head/russian/koi8r-ps
- /ports/head/russian/libreoffice
  /ports/head/russian/mythes
  /ports/head/russian/p5-XML-Parser-encodings
  /ports/head/security/ADMsmb
@@ -1227,8 +1111,6 @@
  /ports/head/textproc/xslide.el
  /ports/head/ukrainian/emacs-koi8u
  /ports/head/ukrainian/hyphen
- /ports/head/ukrainian/libreoffice
- /ports/head/vietnamese/libreoffice
  /ports/head/vietnamese/libviet
  /ports/head/vietnamese/unicode-uhoai
  /ports/head/vietnamese/urwvn
[pro02 dan ~/tmp/homepages] %                                                                     7:46:13

Whereas, stage and prod have only one difference:

[pro02 dan ~/tmp/homepages] % diff -ruN stage prod
--- stage	2022-09-14 07:43:47.000000000 -0400
+++ prod	2022-09-14 07:48:59.000000000 -0400
@@ -560,7 +560,6 @@
  /ports/head/lang/emacs-lisp-intro
  /ports/head/lang/gcc-ecj45
  /ports/head/lang/libobjc2
- /ports/head/lang/linux-dotnet11-sdk
  /ports/head/lang/pbasic
  /ports/head/lang/python-tools
  /ports/head/lang/ratfor
[pro02 dan ~/tmp/homepages] % 

It is time for more investigation.

Website Pin Facebook Twitter Myspace Friendfeed Technorati del.icio.us Digg Google StumbleUpon Premium Responsive