Dan Langille

I've been playing with Open Source software, starting with FreeBSD, since New Zealand Post installed DSL on my street in 1998. From there, I started writing at The FreeBSD Diary, moving my work here after I discovered WordPress. Along the way, I started the BSDCan and PGCon conferences. I slowly moved from software development into full time systems administration and now work for very-well known company who has been a big force in the security industry.

Sep 202022
 

Based on the Add links to GitHub and GitLab mirrors for ports and commits request, the website has some new links arriving soon.

  • Link to GitHub from each commit
  • A similar link to GitLab

Use the one you like. Each git web interface has it’s own followers.

Also, while there, the commit hash at the top of the page is now a link to the commit history further down the page.

EDIT: It seems that link has been present since FreshPorts started processing git commits, and it has always been a link to that commit within cgit.freebsd.org. What changed was the version at the top of the page. That is now a link to #history.

Sep 142022
 

The Libreoffice homepages seem wrong.

dev and test do not show a homepage for arabic/libreoffice, but stage and prod do. Could this be a commit which corrected something after dev and test were corrected?

I think so. I think this commit did that.

Let’s check from the command line.

I’m doing this in my empty jail, which has nothing else in it. I first check out the commit before the one in question. I am using arabic/libreoffice as a reference point.

[empty dan /usr/ports] % sudo git fetch
remote: Enumerating objects: 491805, done.
remote: Counting objects: 100% (99085/99085), done.
remote: Compressing objects: 100% (504/504), done.
remote: Total 491805 (delta 99007), reused 98584 (delta 98581), pack-reused 392720
Receiving objects: 100% (491805/491805), 194.33 MiB | 25.76 MiB/s, done.
Resolving deltas: 100% (285529/285529), completed with 43639 local objects.
From https://git.FreeBSD.org/ports
   e66f56f4af6a..0a071f436dd3  main       -> origin/main
   85e1f363d213..5d6eb9d394c1  2021Q3     -> origin/2021Q3
 * [new branch]                2021Q4     -> origin/2021Q4
 * [new branch]                2022Q1     -> origin/2022Q1
 * [new branch]                2022Q2     -> origin/2022Q2
 * [new branch]                2022Q3     -> origin/2022Q3
 * [new tag]                   11-eol         -> 11-eol
 * [new tag]                   release/12.3.0 -> release/12.3.0
 * [new tag]                   release/13.1.0 -> release/13.1.0
[empty dan /usr/ports] % sudo git checkout 1e0832c71eb3ab4d2a548c5cd799da66013481d2
Updating files: 100% (93162/93162), done.
Note: switching to '1e0832c71eb3ab4d2a548c5cd799da66013481d2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c 

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 1e0832c71eb3 x11-themes/flat-remix-gtk-themes: Update to 20220627
[empty dan /usr/ports] % cd arabic/libreoffice
[empty dan /usr/ports/arabic/libreoffice] % make -V WWW

[empty dan /usr/ports/arabic/libreoffice] % 

Nothing. Let’s try the commit which fixed it.

[empty dan /usr/ports/arabic/libreoffice] % sudo git checkout 9dea2f5dedeb0ef7ce184c85bb97927ed3a8822d
Previous HEAD position was 1e0832c71eb3 x11-themes/flat-remix-gtk-themes: Update to 20220627
HEAD is now at 9dea2f5dedeb editors/libreoffice: pass WWW line to i18n packages too
[empty dan /usr/ports/arabic/libreoffice] % make -V WWW
https://www.libreoffice.org/

Yes, that’s it.

Now, let’s compare this commit against the timing of the correction scripts.

The test correction script finished at about 10:04 AM on Sep 11, 2022. The commit in question was at 2022-09-11 16:42:15 +0000 (or about 12:42 AM). That’s why dev and test are wrong. They didn’t have the fix when the correction script ran.

Your next question might be: why didn’t this get fixed when FreshPorts processed the commit?

There is no standard connection between the commit and the ports it affects. If this was a primary/secondary relationship, as detected by standard methods, for example:

[empty dan /usr/ports/sysutils/bacula13-client] % make -V MASTER_PORT
sysutils/bacula13-server
[empty dan /usr/ports/sysutils/bacula13-client] % 

Rather, the Makefile for arabic/libreoffice is rather brief:

[empty dan /usr/ports/arabic/libreoffice] % cat Makefile
CATEGORIES=	arabic
LO_I18N=	ar
LO_HAS_HELPPACK=	yes

.include "${.CURDIR}/../../editors/libreoffice/Makefile.i18n"

.include 

This is an eloquent solution for a large number of ports. However, the cost to FreshPorts is not knowing to refresh the dependent ports when the primary port is updated.

However, it should be easy to modify my correction script. Let’s try this SQL:

freshports.devgit=#      
  SELECT PA.id,
         PA.category,
         PA.name,
         EP.pathname as port_pathname, homepage
    FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
   WHERE EP.pathname like '/ports/head/%/libreoffice%'
ORDER BY category, name;
  id   |  category  |          name           |                port_pathname                |           homepage           
-------+------------+-------------------------+---------------------------------------------+------------------------------
 30905 | arabic     | libreoffice             | /ports/head/arabic/libreoffice              | 
 30897 | chinese    | libreoffice-zh_CN       | /ports/head/chinese/libreoffice-zh_CN       | 
 30884 | chinese    | libreoffice-zh_TW       | /ports/head/chinese/libreoffice-zh_TW       | 
 28193 | editors    | libreoffice             | /ports/head/editors/libreoffice             | https://www.libreoffice.org/
 30881 | editors    | libreoffice-af          | /ports/head/editors/libreoffice-af          | 
 32344 | editors    | libreoffice-am          | /ports/head/editors/libreoffice-am          | 
 30817 | editors    | libreoffice-as          | /ports/head/editors/libreoffice-as          | 
 31480 | editors    | libreoffice-ast         | /ports/head/editors/libreoffice-ast         | 
 30861 | editors    | libreoffice-be          | /ports/head/editors/libreoffice-be          | 
 30831 | editors    | libreoffice-bg          | /ports/head/editors/libreoffice-bg          | 
 30908 | editors    | libreoffice-bn          | /ports/head/editors/libreoffice-bn          | 
 32343 | editors    | libreoffice-bn_IN       | /ports/head/editors/libreoffice-bn_IN       | 
 31478 | editors    | libreoffice-bo          | /ports/head/editors/libreoffice-bo          | 
 30832 | editors    | libreoffice-br          | /ports/head/editors/libreoffice-br          | 
 30855 | editors    | libreoffice-brx         | /ports/head/editors/libreoffice-brx         | 
 30839 | editors    | libreoffice-bs          | /ports/head/editors/libreoffice-bs          | 
 30837 | editors    | libreoffice-ca          | /ports/head/editors/libreoffice-ca          | 
 34763 | editors    | libreoffice-ca_valencia | /ports/head/editors/libreoffice-ca_valencia | 
 59798 | editors    | libreoffice-ckb         | /ports/head/editors/libreoffice-ckb         | 
 30891 | editors    | libreoffice-cs          | /ports/head/editors/libreoffice-cs          | 
 30904 | editors    | libreoffice-cy          | /ports/head/editors/libreoffice-cy          | 
 30875 | editors    | libreoffice-da          | /ports/head/editors/libreoffice-da          | 
 30911 | editors    | libreoffice-dgo         | /ports/head/editors/libreoffice-dgo         | 
 52543 | editors    | libreoffice-dsb         | /ports/head/editors/libreoffice-dsb         | 
 30894 | editors    | libreoffice-dz          | /ports/head/editors/libreoffice-dz          | 
 30880 | editors    | libreoffice-el          | /ports/head/editors/libreoffice-el          | 
 30863 | editors    | libreoffice-en_GB       | /ports/head/editors/libreoffice-en_GB       | 
 30850 | editors    | libreoffice-en_ZA       | /ports/head/editors/libreoffice-en_ZA       | 
 30907 | editors    | libreoffice-eo          | /ports/head/editors/libreoffice-eo          | 
 30840 | editors    | libreoffice-es          | /ports/head/editors/libreoffice-es          | 
 30912 | editors    | libreoffice-et          | /ports/head/editors/libreoffice-et          | 
 30900 | editors    | libreoffice-eu          | /ports/head/editors/libreoffice-eu          | 
 30866 | editors    | libreoffice-fa          | /ports/head/editors/libreoffice-fa          | 
 30877 | editors    | libreoffice-fi          | /ports/head/editors/libreoffice-fi          | 
 59799 | editors    | libreoffice-fur         | /ports/head/editors/libreoffice-fur         | 
 49014 | editors    | libreoffice-fy          | /ports/head/editors/libreoffice-fy          | 
 30892 | editors    | libreoffice-ga          | /ports/head/editors/libreoffice-ga          | 
 30853 | editors    | libreoffice-gd          | /ports/head/editors/libreoffice-gd          | 
 30830 | editors    | libreoffice-gl          | /ports/head/editors/libreoffice-gl          | 
 30899 | editors    | libreoffice-gu          | /ports/head/editors/libreoffice-gu          | 
 37232 | editors    | libreoffice-gug         | /ports/head/editors/libreoffice-gug         | 
 30829 | editors    | libreoffice-hi          | /ports/head/editors/libreoffice-hi          | 
 30815 | editors    | libreoffice-hr          | /ports/head/editors/libreoffice-hr          | 
 47506 | editors    | libreoffice-hsb         | /ports/head/editors/libreoffice-hsb         | 
 30864 | editors    | libreoffice-i18n        | /ports/head/editors/libreoffice-i18n        | https://www.libreoffice.org/
 30826 | editors    | libreoffice-id          | /ports/head/editors/libreoffice-id          | 
 30842 | editors    | libreoffice-is          | /ports/head/editors/libreoffice-is          | 
 30854 | editors    | libreoffice-it          | /ports/head/editors/libreoffice-it          | 
 30841 | editors    | libreoffice-ka          | /ports/head/editors/libreoffice-ka          | 
 52544 | editors    | libreoffice-kab         | /ports/head/editors/libreoffice-kab         | 
 30914 | editors    | libreoffice-kk          | /ports/head/editors/libreoffice-kk          | 
 30889 | editors    | libreoffice-km          | /ports/head/editors/libreoffice-km          | 
 34762 | editors    | libreoffice-kmr_Latn    | /ports/head/editors/libreoffice-kmr_Latn    | 
 31479 | editors    | libreoffice-kn          | /ports/head/editors/libreoffice-kn          | 
 30903 | editors    | libreoffice-kok         | /ports/head/editors/libreoffice-kok         | 
 30836 | editors    | libreoffice-ks          | /ports/head/editors/libreoffice-ks          | 
 31476 | editors    | libreoffice-lb          | /ports/head/editors/libreoffice-lb          | 
 30868 | editors    | libreoffice-lo          | /ports/head/editors/libreoffice-lo          | 
 30902 | editors    | libreoffice-lt          | /ports/head/editors/libreoffice-lt          | 
 30852 | editors    | libreoffice-lv          | /ports/head/editors/libreoffice-lv          | 
 30828 | editors    | libreoffice-mai         | /ports/head/editors/libreoffice-mai         | 
 30827 | editors    | libreoffice-mk          | /ports/head/editors/libreoffice-mk          | 
 30824 | editors    | libreoffice-ml          | /ports/head/editors/libreoffice-ml          | 
 30893 | editors    | libreoffice-mn          | /ports/head/editors/libreoffice-mn          | 
 30872 | editors    | libreoffice-mni         | /ports/head/editors/libreoffice-mni         | 
 30846 | editors    | libreoffice-mr          | /ports/head/editors/libreoffice-mr          | 
 30874 | editors    | libreoffice-my          | /ports/head/editors/libreoffice-my          | 
 30888 | editors    | libreoffice-nb          | /ports/head/editors/libreoffice-nb          | 
 30885 | editors    | libreoffice-ne          | /ports/head/editors/libreoffice-ne          | 
 30849 | editors    | libreoffice-nl          | /ports/head/editors/libreoffice-nl          | 
 30876 | editors    | libreoffice-nn          | /ports/head/editors/libreoffice-nn          | 
 30865 | editors    | libreoffice-nr          | /ports/head/editors/libreoffice-nr          | 
 30896 | editors    | libreoffice-nso         | /ports/head/editors/libreoffice-nso         | 
 30822 | editors    | libreoffice-oc          | /ports/head/editors/libreoffice-oc          | 
 30895 | editors    | libreoffice-om          | /ports/head/editors/libreoffice-om          | 
 30909 | editors    | libreoffice-or          | /ports/head/editors/libreoffice-or          | 
 30823 | editors    | libreoffice-pa_IN       | /ports/head/editors/libreoffice-pa_IN       | 
 30856 | editors    | libreoffice-ro          | /ports/head/editors/libreoffice-ro          | 
 30857 | editors    | libreoffice-rw          | /ports/head/editors/libreoffice-rw          | 
 30901 | editors    | libreoffice-sa_IN       | /ports/head/editors/libreoffice-sa_IN       | 
 30847 | editors    | libreoffice-sat         | /ports/head/editors/libreoffice-sat         | 
 30890 | editors    | libreoffice-sd          | /ports/head/editors/libreoffice-sd          | 
 30860 | editors    | libreoffice-si          | /ports/head/editors/libreoffice-si          | 
 32520 | editors    | libreoffice-sid         | /ports/head/editors/libreoffice-sid         | 
 30818 | editors    | libreoffice-sk          | /ports/head/editors/libreoffice-sk          | 
 30862 | editors    | libreoffice-sl          | /ports/head/editors/libreoffice-sl          | 
 30870 | editors    | libreoffice-sq          | /ports/head/editors/libreoffice-sq          | 
 30883 | editors    | libreoffice-sr          | /ports/head/editors/libreoffice-sr          | 
 34764 | editors    | libreoffice-sr_Latn     | /ports/head/editors/libreoffice-sr_Latn     | 
 30886 | editors    | libreoffice-ss          | /ports/head/editors/libreoffice-ss          | 
 30843 | editors    | libreoffice-st          | /ports/head/editors/libreoffice-st          | 
 30859 | editors    | libreoffice-sv          | /ports/head/editors/libreoffice-sv          | 
 30835 | editors    | libreoffice-sw_TZ       | /ports/head/editors/libreoffice-sw_TZ       | 
 54774 | editors    | libreoffice-szl         | /ports/head/editors/libreoffice-szl         | 
 30825 | editors    | libreoffice-ta          | /ports/head/editors/libreoffice-ta          | 
 30913 | editors    | libreoffice-te          | /ports/head/editors/libreoffice-te          | 
 30858 | editors    | libreoffice-tg          | /ports/head/editors/libreoffice-tg          | 
 30845 | editors    | libreoffice-th          | /ports/head/editors/libreoffice-th          | 
 30816 | editors    | libreoffice-tn          | /ports/head/editors/libreoffice-tn          | 
 30851 | editors    | libreoffice-tr          | /ports/head/editors/libreoffice-tr          | 
 30844 | editors    | libreoffice-ts          | /ports/head/editors/libreoffice-ts          | 
 31481 | editors    | libreoffice-tt          | /ports/head/editors/libreoffice-tt          | 
 31477 | editors    | libreoffice-ug          | /ports/head/editors/libreoffice-ug          | 
 30873 | editors    | libreoffice-uz          | /ports/head/editors/libreoffice-uz          | 
 30834 | editors    | libreoffice-ve          | /ports/head/editors/libreoffice-ve          | 
 40201 | editors    | libreoffice-vec         | /ports/head/editors/libreoffice-vec         | 
 30879 | editors    | libreoffice-xh          | /ports/head/editors/libreoffice-xh          | 
 30867 | editors    | libreoffice-zu          | /ports/head/editors/libreoffice-zu          | 
 30878 | french     | libreoffice             | /ports/head/french/libreoffice              | 
 30898 | german     | libreoffice             | /ports/head/german/libreoffice              | 
 30819 | hebrew     | libreoffice             | /ports/head/hebrew/libreoffice              | 
 30910 | hungarian  | libreoffice             | /ports/head/hungarian/libreoffice           | 
 30820 | japanese   | libreoffice             | /ports/head/japanese/libreoffice            | 
 30838 | korean     | libreoffice             | /ports/head/korean/libreoffice              | 
 30848 | polish     | libreoffice             | /ports/head/polish/libreoffice              | 
 30906 | portuguese | libreoffice             | /ports/head/portuguese/libreoffice          | 
 30869 | portuguese | libreoffice-pt_BR       | /ports/head/portuguese/libreoffice-pt_BR    | 
 30833 | russian    | libreoffice             | /ports/head/russian/libreoffice             | 
 30821 | ukrainian  | libreoffice             | /ports/head/ukrainian/libreoffice           | 
 30814 | vietnamese | libreoffice             | /ports/head/vietnamese/libreoffice          | 
(120 rows)
freshports.devgit=#     

After running the correction script again, this time only upon Librenms ports, dev and stage match.

Similarly, running the fix on test resulted in identical results on both dev, test, and stage.

Now it’s just lang/linux-dotnet11-sdk which differs. This also seems to be a case of bad timing. This commit seems to have changed that port

Running a refresh on that one port (lang/linux-dotnet11-sdk) on each of dev, test, and stage results in a match amongst all four websites.

Done.

Sep 142022
 

By “post correction”, I meant “post-correction script”, i.e. after the correction script completed.

A recent commit started the process of moving the WWW from pkg-descr to Makefile. These were huge commits, about 27930 ports each. FreshPorts is dumb. It sees a commit and it refreshes all ports touched by that commit. Refreshing every port takes a long time. It’s not just running make -V and extracting values. It looks at DEPENDS, pkg-plist, refreshes secondary ports based on changes to primary ports. Some ports take a second to do this. Some take 5-7 seconds. Even at 1 second per port, we’re looking at about 8 hours to process these commits.

That’s not the big issue. The bigger issue was the clearing out of all existing homepage values. With no WWW entry in pkg-descr, FreshPorts set that value as emtpy in the database.

The solution: update the code to use make -V WWW and if empty, take the value from pkg-descr. This “if empty” option would allow quarterly branch ports to continue to work.

The diff for the code change is not big.

With new code packaged and a refresh-ports scripts ready to go, I started rolling the changes out to dev, test, stage, and prod. This whole process took about about a week, based on this first tweet.

The results

[/sourcecode]

The purpose of this blog post is to record any differences between the various FreshPorts databases in case that become relevant later.

The goal: how many [active] ports have empty homepages? By active, I mean non-deleted ports; ports you would find in a git clone output.

dev

On dev, we have 1465 such ports.

freshports.devgit=#   SELECT count(*)
    FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
  AND (PA.homepage is null or trim(PA.homepage) = '');
 count 
-------
  1465
(1 row)


<h3>test</h3>

On <span class="cmd">test</span>, we have 1465 such ports.


freshports.testgit=#   SELECT count(*)
    FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
  AND (PA.homepage is null or trim(PA.homepage) = '');
 count 
-------
  1465
(1 row)


<h3>stage</h3>

On <span class="cmd">stage</span>, we have 1347 such ports.


You are now connected to database "freshports.stagegit" as user "dan".
freshports.stagegit=#   SELECT count(*)
    FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
  AND (PA.homepage is null or trim(PA.homepage) = '');
 count 
-------
  1347
(1 row)


<h3>prod</h3>

On <span class="cmd">prod</span>, we have 1465 such ports.


freshports.devgit=#   SELECT count(*)
    FROM ports_active PA JOIN element_pathname EP on PA.element_id = EP.element_id
where EP.pathname like '/ports/head/%'
  AND (PA.homepage is null or trim(PA.homepage) = '');
 count 
-------
  1465
(1 row)


<h2>What's next?</h2>

I want to see the differences between these results. Which ports are different? That shouldn't take much. A short script to pull out the results. A few diff statements.

<h3>dev vs stage</h3>

The results have been captured into files:

<pre class="screen">
[pro02 dan ~/tmp/homepages] % wc -l *
    1464 dev
    1345 prod
    1346 stage
    1464 test
    5619 total
</pre>

It looks like <span class="file">dev</span> and <span class="file">test</span> match:


[pro02 dan ~/tmp/homepages] % diff -ruN dev test
[pro02 dan ~/tmp/homepages] % 

The big diff is dev vs stage and seem related to libreoffice:

[pro02 dan ~/tmp/homepages] % diff -ruN dev stage
--- dev	2022-09-14 07:45:03.000000000 -0400
+++ stage	2022-09-14 07:43:47.000000000 -0400
@@ -2,7 +2,6 @@
  /ports/head/arabic/ae_fonts_mono
  /ports/head/arabic/ae_fonts_ttf
  /ports/head/arabic/kacst_fonts
- /ports/head/arabic/libreoffice
  /ports/head/archivers/deb2targz
  /ports/head/archivers/ha
  /ports/head/archivers/lcab
@@ -87,8 +86,6 @@
  /ports/head/chinese/hc
  /ports/head/chinese/jis2gb
  /ports/head/chinese/kcfonts
- /ports/head/chinese/libreoffice-zh_CN
- /ports/head/chinese/libreoffice-zh_TW
  /ports/head/chinese/lunar
  /ports/head/chinese/moettf
  /ports/head/chinese/ttfm
@@ -312,109 +309,6 @@
  /ports/head/editors/edith
  /ports/head/editors/hexpert
  /ports/head/editors/jove
- /ports/head/editors/libreoffice-af
- /ports/head/editors/libreoffice-am
- /ports/head/editors/libreoffice-as
- /ports/head/editors/libreoffice-ast
- /ports/head/editors/libreoffice-be
- /ports/head/editors/libreoffice-bg
- /ports/head/editors/libreoffice-bn
- /ports/head/editors/libreoffice-bn_IN
- /ports/head/editors/libreoffice-bo
- /ports/head/editors/libreoffice-br
- /ports/head/editors/libreoffice-brx
- /ports/head/editors/libreoffice-bs
- /ports/head/editors/libreoffice-ca
- /ports/head/editors/libreoffice-ca_valencia
- /ports/head/editors/libreoffice-ckb
- /ports/head/editors/libreoffice-cs
- /ports/head/editors/libreoffice-cy
- /ports/head/editors/libreoffice-da
- /ports/head/editors/libreoffice-dgo
- /ports/head/editors/libreoffice-dsb
- /ports/head/editors/libreoffice-dz
- /ports/head/editors/libreoffice-el
- /ports/head/editors/libreoffice-en_GB
- /ports/head/editors/libreoffice-en_ZA
- /ports/head/editors/libreoffice-eo
- /ports/head/editors/libreoffice-es
- /ports/head/editors/libreoffice-et
- /ports/head/editors/libreoffice-eu
- /ports/head/editors/libreoffice-fa
- /ports/head/editors/libreoffice-fi
- /ports/head/editors/libreoffice-fur
- /ports/head/editors/libreoffice-fy
- /ports/head/editors/libreoffice-ga
- /ports/head/editors/libreoffice-gd
- /ports/head/editors/libreoffice-gl
- /ports/head/editors/libreoffice-gu
- /ports/head/editors/libreoffice-gug
- /ports/head/editors/libreoffice-hi
- /ports/head/editors/libreoffice-hr
- /ports/head/editors/libreoffice-hsb
- /ports/head/editors/libreoffice-id
- /ports/head/editors/libreoffice-is
- /ports/head/editors/libreoffice-it
- /ports/head/editors/libreoffice-ka
- /ports/head/editors/libreoffice-kab
- /ports/head/editors/libreoffice-kk
- /ports/head/editors/libreoffice-km
- /ports/head/editors/libreoffice-kmr_Latn
- /ports/head/editors/libreoffice-kn
- /ports/head/editors/libreoffice-kok
- /ports/head/editors/libreoffice-ks
- /ports/head/editors/libreoffice-lb
- /ports/head/editors/libreoffice-lo
- /ports/head/editors/libreoffice-lt
- /ports/head/editors/libreoffice-lv
- /ports/head/editors/libreoffice-mai
- /ports/head/editors/libreoffice-mk
- /ports/head/editors/libreoffice-ml
- /ports/head/editors/libreoffice-mn
- /ports/head/editors/libreoffice-mni
- /ports/head/editors/libreoffice-mr
- /ports/head/editors/libreoffice-my
- /ports/head/editors/libreoffice-nb
- /ports/head/editors/libreoffice-ne
- /ports/head/editors/libreoffice-nl
- /ports/head/editors/libreoffice-nn
- /ports/head/editors/libreoffice-nr
- /ports/head/editors/libreoffice-nso
- /ports/head/editors/libreoffice-oc
- /ports/head/editors/libreoffice-om
- /ports/head/editors/libreoffice-or
- /ports/head/editors/libreoffice-pa_IN
- /ports/head/editors/libreoffice-ro
- /ports/head/editors/libreoffice-rw
- /ports/head/editors/libreoffice-sa_IN
- /ports/head/editors/libreoffice-sat
- /ports/head/editors/libreoffice-sd
- /ports/head/editors/libreoffice-si
- /ports/head/editors/libreoffice-sid
- /ports/head/editors/libreoffice-sk
- /ports/head/editors/libreoffice-sl
- /ports/head/editors/libreoffice-sq
- /ports/head/editors/libreoffice-sr
- /ports/head/editors/libreoffice-sr_Latn
- /ports/head/editors/libreoffice-ss
- /ports/head/editors/libreoffice-st
- /ports/head/editors/libreoffice-sv
- /ports/head/editors/libreoffice-sw_TZ
- /ports/head/editors/libreoffice-szl
- /ports/head/editors/libreoffice-ta
- /ports/head/editors/libreoffice-te
- /ports/head/editors/libreoffice-tg
- /ports/head/editors/libreoffice-th
- /ports/head/editors/libreoffice-tn
- /ports/head/editors/libreoffice-tr
- /ports/head/editors/libreoffice-ts
- /ports/head/editors/libreoffice-tt
- /ports/head/editors/libreoffice-ug
- /ports/head/editors/libreoffice-uz
- /ports/head/editors/libreoffice-ve
- /ports/head/editors/libreoffice-vec
- /ports/head/editors/libreoffice-xh
- /ports/head/editors/libreoffice-zu
  /ports/head/editors/nvi-m17n
  /ports/head/editors/sted
  /ports/head/editors/uemacs
@@ -429,7 +323,6 @@
  /ports/head/emulators/vxtools
  /ports/head/finance/p5-Business-OnlinePayment-2CheckOut
  /ports/head/finance/p5-Business-OnlinePayment-PaymentsGateway
- /ports/head/french/libreoffice
  /ports/head/ftp/fmirror
  /ports/head/ftp/ftpfind
  /ports/head/ftp/ftpmirror
@@ -555,7 +448,6 @@
  /ports/head/german/aspell
  /ports/head/german/eric6
  /ports/head/german/hyphen
- /ports/head/german/libreoffice
  /ports/head/german/phone
  /ports/head/graphics/4va
  /ports/head/graphics/bmp2html
@@ -593,9 +485,7 @@
  /ports/head/graphics/xoris
  /ports/head/graphics/xtexcad
  /ports/head/graphics/xwpick
- /ports/head/hebrew/libreoffice
  /ports/head/hungarian/hyphen
- /ports/head/hungarian/libreoffice
  /ports/head/hungarian/mythes
  /ports/head/irc/bnc
  /ports/head/irc/irc
@@ -627,7 +517,6 @@
  /ports/head/japanese/kinput2
  /ports/head/japanese/kon2-14dot
  /ports/head/japanese/kon2-16dot
- /ports/head/japanese/libreoffice
  /ports/head/japanese/libslang
  /ports/head/japanese/man
  /ports/head/japanese/man-doc
@@ -655,7 +544,6 @@
  /ports/head/korean/hmconv
  /ports/head/korean/hpscat
  /ports/head/korean/kaistfonts
- /ports/head/korean/libreoffice
  /ports/head/korean/munhwafonts-cid
  /ports/head/korean/nanum-barun-gothic
  /ports/head/korean/nanum-barun-pen
@@ -935,7 +823,6 @@
  /ports/head/news/trn
  /ports/head/polish/fortunepl
  /ports/head/polish/hyphen
- /ports/head/polish/libreoffice
  /ports/head/ports-mgmt/genpatch
  /ports/head/ports-mgmt/lsknobs
  /ports/head/ports-mgmt/pchecker
@@ -954,8 +841,6 @@
  /ports/head/ports-mgmt/wanted-ports
  /ports/head/portuguese/aspell-pt_PT
  /ports/head/portuguese/hyphen
- /ports/head/portuguese/libreoffice
- /ports/head/portuguese/libreoffice-pt_BR
  /ports/head/print/afm
  /ports/head/print/asprint
  /ports/head/print/bibcursed
@@ -1002,7 +887,6 @@
  /ports/head/russian/hyphen
  /ports/head/russian/koi2koi
  /ports/head/russian/koi8r-ps
- /ports/head/russian/libreoffice
  /ports/head/russian/mythes
  /ports/head/russian/p5-XML-Parser-encodings
  /ports/head/security/ADMsmb
@@ -1227,8 +1111,6 @@
  /ports/head/textproc/xslide.el
  /ports/head/ukrainian/emacs-koi8u
  /ports/head/ukrainian/hyphen
- /ports/head/ukrainian/libreoffice
- /ports/head/vietnamese/libreoffice
  /ports/head/vietnamese/libviet
  /ports/head/vietnamese/unicode-uhoai
  /ports/head/vietnamese/urwvn
[pro02 dan ~/tmp/homepages] %                                                                     7:46:13

Whereas, stage and prod have only one difference:

[pro02 dan ~/tmp/homepages] % diff -ruN stage prod
--- stage	2022-09-14 07:43:47.000000000 -0400
+++ prod	2022-09-14 07:48:59.000000000 -0400
@@ -560,7 +560,6 @@
  /ports/head/lang/emacs-lisp-intro
  /ports/head/lang/gcc-ecj45
  /ports/head/lang/libobjc2
- /ports/head/lang/linux-dotnet11-sdk
  /ports/head/lang/pbasic
  /ports/head/lang/python-tools
  /ports/head/lang/ratfor
[pro02 dan ~/tmp/homepages] % 

It is time for more investigation.

Sep 102022
 

Today I learned, or at best, relearned, a new make target:

root@freshports:/usr/ports/sysutils/anvil # make create-manifest
root@freshports:/usr/ports/sysutils/anvil # ls -lt | head
total 14
drwxr-xr-x  3 root  wheel     3 Sep 10 17:26 work
-rw-r--r--  1 root  wheel   337 Sep 10 16:06 pkg-descr
-rw-r--r--  1 root  wheel   182 Sep 10 16:06 distinfo
-rw-r--r--  1 root  wheel  1050 Sep 10 16:06 Makefile
-rw-r--r--  1 root  wheel   146 Aug 12 14:36 pkg-plist
drwxr-xr-x  2 root  wheel     6 Aug 12 14:36 files
root@freshports:/usr/ports/sysutils/anvil # find work
work
work/.metadir
work/.metadir/+MANIFEST
work/.metadir/+DISPLAY
work/.metadir/+DESC

That first file contains many things which FreshPorts now obtains via make -V:

root@freshports:/usr/ports/sysutils/anvil # cat work/.metadir/+MANIFEST 
name: "anvil"
version: "0.0.20"
origin: sysutils/anvil
comment: <<EOD
Tools for distributing ssl certificates
EOD
maintainer: dvl@FreeBSD.org
prefix: /usr/local
categories: [ sysutils, ]
licenselogic: single
www: "https://github.com/dlangille/anvil"
licenses: [ BSD2CLAUSE, ]
users: [ anvil, ]
groups: [ anvil, ]
deps: { 
}
options: {
 FETCH: on,
 CURL: off,
 WGET: off,
}
root@freshports:/usr/ports/sysutils/anvil # make -V LICENSE
BSD2CLAUSE

It might also simplify the parsing it now does for pkg-descr:

root@freshports:/usr/ports/sysutils/anvil # cat ./work/.metadir/+DISPLAY
[
]
root@freshports:/usr/ports/sysutils/anvil # cat ./work/.metadir/+DESC
Tools for distributing ssl certificates

Designed for FreeBSD (it uses fetch, not wget or curl [yet]).

It also uses sudo, with the goal of this running as non-root
and only allowing the cp & mv via sudo.

These tools were designed with acme.sh & Let's Encrypt in mind,
but they should with with any certificates generated by any
means.

WWW: https://github.com/dlangille/anvil
root@freshports:/usr/ports/sysutils/anvil # 

This doubly true for slave ports:

root@freshports:/usr/ports/sysutils/anvil # cd ../bacula13-client
root@freshports:/usr/ports/sysutils/bacula13-client # make create-manifest
root@freshports:/usr/ports/sysutils/bacula13-client # find .
.
./Makefile
./work
./work/.metadir
./work/.metadir/+DESC
./work/.metadir/+DISPLAY
./work/.metadir/+MANIFEST
root@freshports:/usr/ports/sysutils/bacula13-client # cat ./work/.metadir/+DESC
Bacula is a set of computer programs that permit you (or the system
administrator) to manage backup, recovery, and verification of
computer data across a network of computers of different kinds.
In technical terms, it is a network Client/Server based backup program.
Bacula is relatively easy to use and efficient, while offering many
advanced storage management features that make it easy to find and
recover lost or damaged files. Due to its modular design, Bacula is
scalable from small single computer systems to systems consisting of
hundreds of computers located over a large network.

WWW: https://www.bacula.org/
root@freshports:/usr/ports/sysutils/bacula13-client # cat ./work/.metadir/+DISPLAY
[
]
root@freshports:/usr/ports/sysutils/bacula13-client # cat ./work/.metadir/+MANIFEST
name: "bacula13-client"
version: "13.0.1"
origin: sysutils/bacula13-client
comment: <<EOD
Network backup solution (client)
EOD
maintainer: dvl@FreeBSD.org
prefix: /usr/local
categories: [ sysutils, ]
licenselogic: single
www: "https://www.bacula.org/"
licenses: [ AGPLv3, ]
users: [ bacula, ]
groups: [ bacula, ]
deps: { 
}
options: {
 IPV6: on,
 NLS: on,
 PYTHON: off,
}
annotations: {
 cpe: <<EOD
cpe:2.3:a:bacula:bacula:13.0.1:::::freebsd13:x64
EOD
}
Aug 192022
 

A long-standing bug, perhaps dating back to April 2003, has been fixed. It was an optimization in a trigger: if the values in the trigger have not changed, let’s not run the trigger.

Turns out, that was a bad idea.

The table

The ports_categories table documents the connections between a port and the categories in which it resides. There must be one physical category. By physical, I mean a category for which a directory exists on disk in the /usr/ports directory. This is sometimes referred to as the primary category. The primary/physical category must always be listed first in the CATEGORIES= directive of the port’s Makefile. See also Categorization in the FreeBSD Porters Handbook.

There are four open issues as I write this. The missing ports_categories entries #129 issue is most relevant to what I’m trying to solve today.

How I know

A system check is run on on regular basis and I get an email looking like this:

This is a list of ports that do not have entries in the ports_categories table
This can be fixed with this query:

begin;  insert into ports_categories select id, category_id from ports_active PA WHERE NOT EXISTS 
(SELECT * from ports_categories PC where PC.port_id = PA.id and PC.category_id = PA.category_id);

This is a list of the ports in question:

 id  | category_id |       name       |   category   |             port              |             element_pathname              
-----+-------------+------------------+--------------+-------------------------------+-------------------------------------------
 500 |           9 | open-motif-devel | x11-toolkits | x11-toolkits/open-motif-devel | /ports/head/x11-toolkits/open-motif-devel
(1 row)

Checking the data

Looking at the history of commits for that port, I notice the most recent commit resurrected this port after it was deleted in 2002.

I now suspect the resurrection process on FreshPorts. Perhaps the updates to ports_categories are not being done.

Let’s look at the data:

[pg02 dan ~] % psql freshports.devgit
psql (12.11)
Type "help" for help.

freshports.devgit=# select category_id, categories from ports where id = 500;
 category_id |  categories  
-------------+--------------
           9 | x11-toolkits
(1 row)

freshports.devgit=# select * from ports_categories where port_id = 500;
 port_id | category_id 
---------+-------------
(0 rows)

freshports.devgit=# 

From there, I can see category_id (derived value) and categories (directly copied from the Makefile are populated. It seems the only information missed is the entry from the ports_categories table.

The files

Here are the files/logs involved with that commit. They are archived by commit date.

[dev-ingress01 dan ~freshports/message-queues/recent] % cd ../archive/2022_08/2022_08_16
[dev-ingress01 dan ~freshports/message-queues/archive/2022_08/2022_08_16] % ls *0395dfd3904676e2e652bfd7850c4a48ec5cebba*
2022.08.16.03.33.52.000000.0395dfd3904676e2e652bfd7850c4a48ec5cebba.log  2022.08.16.03.33.52.000000.0395dfd3904676e2e652bfd7850c4a48ec5cebba.xml
[dev-ingress01 dan ~freshports/message-queues/archive/2022_08/2022_08_16] %

Can it be reproduced?

I have an idea. Let’s rerun the commit and see if the situation persists. If so, we have a reproducible event.

This is the delete:

freshports.devgit=# begin; delete from commit_log where message_id = '0395dfd3904676e2e652bfd7850c4a48ec5cebba';
BEGIN
DELETE 1
freshports.devgit=# commit;
COMMIT
freshports.devgit=# 

This is the injection of the commit into the processing queue:

[dev-ingress01 dan ~freshports/message-queues/archive/2022_08/2022_08_16] % sudo mv -i 2022.08.16.03.33.52.000000.0395dfd3904676e2e652bfd7850c4a48ec5cebba.xml ~ingress/message-queues/incoming/ 

The files are now back in the recent queue and this is how we know processing is complete.

[dev-ingress01 dan ~freshports/message-queues/recent] % ls *0395dfd3904676e2e652bfd7850c4a48ec5cebba*
2022.08.16.03.33.52.000000.0395dfd3904676e2e652bfd7850c4a48ec5cebba.log  2022.08.16.03.33.52.000000.0395dfd3904676e2e652bfd7850c4a48ec5cebba.xml

Checking the tables, we have the same results. The situation can be reproduced.

freshports.devgit=# select category_id, categories from ports where id = 500;
 category_id |  categories  
-------------+--------------
           9 | x11-toolkits
(1 row)

freshports.devgit=# select * from ports_categories where port_id = 500;
 port_id | category_id 
---------+-------------
(0 rows)

freshports.devgit=# 

I looked at the code, and found nothing updating the ports_categories table. It must be part of the database itself (i.e. a trigger).

Looking at the source code for the relational integrity stuff, I found this function:

CREATE OR REPLACE FUNCTION ports_categories_set() RETURNS TRIGGER AS $$
   DECLARE
                CategoryCount   integer;
                CategoryNames   text;
                Category        text;
                CategoryID      int8;
                UpdateNeeded    boolean;
   BEGIN
                UpdateNeeded := TRUE;

                IF TG_OP = 'UPDATE' THEN
                        IF old.categories = new.categories THEN
                                UpdateNeeded := FALSE;
                        END IF;
                END IF;

                IF UpdateNeeded THEN

                        DELETE 
                          FROM ports_categories
                         WHERE port_id = new.id;

                        IF new.categories IS NOT NULL THEN

                                CategoryCount := 0;
                                CategoryNames := new.categories;
                                LOOP
...

The highlighted lines 12-14 are an optimization. I think it’s wrong. Presumably the table would be correct if no changes are occurring. However, I see the faulty logic and it’s not in the function above.

That code dates back to at least April 2003.

Background

When the ports_categories table was first added to the database, it needed to be populated. Looking at commit logs, this was sometime pre 2006-12-17 because the oldest log entry says:

r4012 | dan | 2006-12-17 12:06:34 +0000 (Sun, 17 Dec 2006) | 2 lines

Merge from FreshPorts2 to HEAD

I could not find any earlier references. In that populate script, I find:

#
# get a list of ports to update
#

$sql = "
  SELECT ports_active.id         AS id, 
         ports_active.categories AS categories,
         ports_active.name       AS name,
         ports_active.category   AS category
    FROM ports_active
ORDER BY ports_active.id";

The ports_active is a view of the ports table which excludes deleted ports.

That’s the problem right there ^

Why were deleted ports excluded? It is awkward to extract values from deleted ports. The Makefile may be out of data for the current FreshPorts code. Sure, the code could versioned and could cope with older Makefile – that approach is common with APIs. No, I’m not going to do that.

Let’s comment out that optimization from the trigger and see if that fixes the situation.

Checking

Not shown: editing the trigger, updating the database, deleting the commit, and injecting the XML file back into the processing queue.

Here’s the new situation:

freshports.devgit=# select * from ports_categories where port_id = 500;
 port_id | category_id 
---------+-------------
     500 |           9
(1 row)

freshports.devgit=# select category_id, categories from ports where id = 500;
 category_id |  categories  
-------------+--------------
           9 | x11-toolkits
(1 row)

That is correct. The ports_categories now contains the expected values. Good.

I still have to wait for the top of the hour when the system check will be invoked, but right now, I’m confident this has solved an elusive bug.

EDIT: Several hours later, the alerts are no longer occurring on the dev database. Time to push these change through to test, stage, and then prod.

Jul 092022
 

The July 4th tweet announced “dev, test, and stage .freshports.org are now running Python 3.9”. Three days later, “Is it coincidental that dev, test, and stage are all stuck on commits from July 4?”

I took notes as I explored the problem. The first clue was:


2022.07.07 00:12:03 git-delta.sh /usr/local/libexec/freshports/git-to-freshports-xml.py –repo ports –path /var/db/ingress/repos/ports –branch main –commit-range e92af65a1220ad4e3a567f67e3ab0ff259910631..271ce919763ea8e3bad7f642f13e4cc13fbad5ce –spooling /var/db/ingress/message-queues/spooling –output /var/db/ingress/message-queues/incoming
/usr/local/libexec/freshports/git-delta.sh: /usr/local/libexec/freshports/git-to-freshports-xml.py: not found

That file exists:

[dev-ingress01 dan /var/log/freshports] % pkg which /usr/local/libexec/freshports/git-to-freshports-xml.py                                                                                                                         19:27:27
/usr/local/libexec/freshports/git-to-freshports-xml.py was installed by package py39-freshports-git-proc-commit-0.1.1

The problem was the shebang:

[dev-ingress01 dan /var/log/freshports] % head /usr/local/libexec/freshports/git-to-freshports-xml.py                                                                                                                              19:27:46
#!/usr/local/bin/python3.8


Yeah, I shouldn't be doing that.

The code was fixed. Later, I modified the code to raise alarms which would be noticed.

Catching up on missing code

The code is now catching up on commits made after July 7th, because that’s when commit processing was manually stopped. Next, the system needs to process commits between July 4th and July 7th.

When a commits are processed, the hash of the last commit found is stored as a tag in the local copy of the repo. This tag is used as a reference when processing next occurs. It is the starting point. The old code was updating the even though processing failed to create the required XML file[s]. The new code does not not make this error.

My job now is is locate the hash of the missing code and reset it.

Looking at the data

Here is the SQL I ran. I should put this into a page on FreshPorts. It might be useful.

freshports.devgit=# select * from repo;
 id | name  |      description       |   repo_hostname    | path_to_repo | repository | db_root_prefix 
----+-------+------------------------+--------------------+--------------+------------+----------------
  1 | ports | The FreeBSD Ports tree | svnweb.freebsd.org | /ports       | subversion | 
  2 | doc   | The FreeBSD doc tree   | svnweb.freebsd.org | /doc         | subversion | 
  3 | src   | The FreeBSD src tree   | svnweb.freebsd.org | /base        | subversion | 
  9 | src   | The FreeBSD src tree   | cgit.freebsd.org   | /src         | git        | /base
  8 | doc   | The FreeBSD doc tree   | cgit.freebsd.org   | /doc         | git        | /doc
  7 | ports | The FreeBSD Ports tree | cgit.freebsd.org   | /ports       | git        | /ports
(6 rows)


freshports.devgit=# select id, message_id from commit_log where repo_id = 7 order by message_date desc limit 1;
   id   |                message_id                
--------+------------------------------------------
 911534 | db5eeb201b93c19c42724ec98f8ca1be9e88db15
(1 row)

freshports.devgit=# select id, message_id from commit_log where repo_id = 8 order by message_date desc limit 1;
   id   |                message_id                
--------+------------------------------------------
 911515 | 21de22b400a3970b2d3d4b2831c44702698f57e7
(1 row)

freshports.devgit=# select id, message_id from commit_log where repo_id = 9 order by message_date desc limit 1;
   id   |                message_id                
--------+------------------------------------------
 911537 | 03473e8ec8fa8d0f1ea30f85d8796ea9bf94bf29
(1 row)

freshports.devgit=#

I ran similar queries on test and stage to see how that works. They had different id values for repo_id, but the results were the same. I expect this, but verified it. This means I can run the same git tag commands on each database instance.

I know that I can do that in one query, but I can’t recall how. I have it in code somewhere.

Confirming the dates

I confirmed the dates of the above commits and compared it against the Python upgrade.

  1. src: 03473e8ec8fa8d0f1ea30f85d8796ea9bf94bf29 2022.07.04 20:45:04
  2. doc: 21de22b400a3970b2d3d4b2831c44702698f57e7 2022.07.04 16:51:02
  3. ports: db5eeb201b93c19c42724ec98f8ca1be9e88db15 2022.07.04 20:30:03

When was Python updated?

[dev-ingress01 dan /usr/local/libexec/freshports] % grep python39 /var/log/messages
Jul  4 20:49:21 dev-ingress01 pkg[66393]: python39-3.9.13 installed

Those dates look right.

Where I started to understand the tag

This is where I got confused and was not sure of the exact approach. The code did this


git tag -m “last known commit of ${refname}” -f freshports/${refname} ${refname}

The confusing part was parsing the above:


git tag -m “last known commit of main/origin” -f freshports/origin/main origin/main

I expected this command to take a commit hash. In a way, it is. It’s saying: the last commit I processed is the most recent commit (the one at the top).

Ahh! OK, but I can also supply a hash.

Let’s review the command:

  1. -m Use the given tag message
  2. -f Replace an existing tag with the given name (instead of failing)

What this does: it tags the last commit with freshports/origin/main.

The code uses that later. See my annotation of the code where I explain the code to myself. See also my attempt to find an online code reviewer and my earlier misunderstanding and reading of that code. That confusion lasted about 2 hours but I’m much better for it now.

Setting the tag

Let’s do it for doc first, because that’s a much lower volume repo.

Setting this will cause duplicate commits to be processed, but if a commit is already in FreshPorts, it is not reloaded into the database. I could invoke git-to-freshports-xml.py with a range, but I’d rather do it this way.

This would be:


git tag -m “last known commit of main/origin” -f freshports/origin/main 21de22b400a3970b2d3d4b2831c44702698f57e7

That hash value is from an earlier section.

[dev-ingress01 dan ~ingress/repos/doc] % echo 'git tag -m "last known commit of main/origin" -f freshports/origin/main 21de22b400a3970b2d3d4b2831c44702698f57e7' | sudo su -fm ingress
Updated tag 'freshports/origin/main' (was 6a70cd2969)

Now I wait…

There we go:

[dev-ingress01 dan ~ingress/repos/ports] % sudo xtail /var/log/freshports

*** /var/log/freshports/freshports.log ***
Jul  9 16:51:00 dev-ingress01 FreshPorts[10344]: into /usr/local/etc/periodic/everythreeminutes/100.fp_system_status
Jul  9 16:51:00 dev-ingress01 FreshPorts[10392]: into /usr/local/etc/periodic/everythreeminutes/215.fp_check_git_for_commits

*** /var/log/freshports/ingress-daemon.log ***
Jul  9 16:51:00 dev-ingress01 check_for_git_commits.sh[10398]: touching ~ingress/signals/check_git ~ingress/signals/job_waiting
Jul  9 16:51:00 dev-ingress01 check_for_git_commits.sh[10428]: done touching, going away now
Jul  9 16:51:02 dev-ingress01 ingress[71055]: yes, there is a job waiting
Jul  9 16:51:02 dev-ingress01 ingress[71055]: running /usr/local/bin/perl ./job-waiting.pl
Jul  9 16:51:02 dev-ingress01 ingress[71055]: from directory  /usr/local/libexec/freshports
Jul  9 16:51:02 dev-ingress01 ingress[71055]: -rw-r--r--  1 dan  dan  2571 Apr 24 18:40 ./job-waiting.pl

*** /var/log/freshports/freshports.log ***
Jul  9 16:51:03 dev-ingress01 FreshPorts[11369]: running job-waiting.pl 
Jul  9 16:51:03 dev-ingress01 FreshPorts[11369]: starting ./job-waiting.pl (/usr/local/libexec/freshports) 
Jul  9 16:51:03 dev-ingress01 FreshPorts[11369]: running ./job-waiting.pl as user = 'ingress' (/usr/local/libexec/freshports) 
Jul  9 16:51:03 dev-ingress01 FreshPorts[11369]: checking jobs for ingress (/usr/local/libexec/freshports) 
Jul  9 16:51:03 dev-ingress01 FreshPorts[11369]: /var/db/ingress/signals/check_git exists.  About to run check_git.sh (/usr/local/libexec/freshports) 
Jul  9 16:51:03 dev-ingress01 check_git.sh[11437]: /usr/local/libexec/freshports/check_git.sh has started
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '16f7503065af6905cf0c89f6bbe0c5f4e22d7058 website/themes/beastie/i18n/fr.toml: Fix some typos'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit 'd0e5718b926292f7e85b47a28c05fe7f5a143554 website/content/fr/where.adoc: Fix translation'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '21b5dd496e13ab140eee747d64e9dc4b477a222c - traditional Chinese Translation of the news items (March 2022)'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '5286308fc42391f43fa675580dd67e12521490bb fix build'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '763a2c6fbdf6f219a8c4d2880c25212d3af7467f - traditional Chinese Translation of the news items (April 2022)'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '19586a4dd1e7c2b65f9563e7c49c04814f8ab153 - traditional Chinese Translation of the news items (May 2022)'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '47dcae406e2e0ee34e3766dd9ee04ae4d10642b8 - traditional Chinese Translation of the news items (June 2002)'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit 'a6a3fc9c74a1a055bff161440220ca819caa15a8 fix typo and an untranslated item.'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit 'f814150c1af932259a239ad01c0237f58bdce402 New committer (doc): Graham Perrin, grahamperrin@'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '3a4370a41f988ad0b30f1f256c6ac73aa6e9b248 - traditional Chinese Translation of the latest news item (2022-06-01)'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit 'd9b55a60fba211836e97d39920c55a83cc4443b6 Update information about the clusteradm team'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit 'e321dc3829ee30d5eebdd4d14027dba139d85fc1 [doc-es][articles/contributors]: Keep up with latest changes'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '346a108dfb5f7bbfc2ccbe844c6a6a1aff449442 [doc-es][articles/contributing] Keep up with latest changes'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '43995b87b0ab9b5250283f5338ba824ee9bf32a8 Document FreeBSD version 1400062 and 1400063'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '8ae6611622a426406377d55a0b75ad9965a74ac2 Document FreeBSD version 1301505'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '393a57488dec89f58455ffdb9d3f94dcb03bf080 Document FreeBSD version 1203506 and 1203507'
Jul  9 16:51:03 dev-ingress01 git-to-freshports-xml.py[11506]:[INFO] Processing commit '31fe9dc1198f997305ace049b055254007fc7963 Update CORE Team liaison'

Checking… that first commit is from 4 July.

Looking in the logs, here’s one which was processed but already in the database:

[dev-ingress01 dan ~freshports/message-queues/recent] % cat 2022.07.08.20.04.11.000016.31fe9dc1198f997305ace049b055254007fc7963.log                                                      16:53:36
grabbing categories from disk
'accessibility arabic archivers astro audio base benchmarks biology cad chinese comms converters databases deskutils devel distfiles dns editors emulators finance french ftp games german graphics hebrew hungarian irc japanese java korean lang mail math misc multimedia net net-im net-mgmt net-p2p news polish ports-mgmt portuguese print russian science security shells sysutils textproc ukrainian vietnamese www x11 x11-clocks x11-drivers x11-fm x11-fonts x11-servers x11-themes x11-toolkits x11-wm'
categories already loaded
about to process
finished setting up the Parser
Processing file [/var/db/ingress/message-queues/incoming/2022.07.08.20.04.11.000016.31fe9dc1198f997305ace049b055254007fc7963.xml]...
parsing file now


 *** start of all updates ***


 *** end of all updates *** 
source in getSource is 'git'
That commit is of Type: 'git'
EOF
invoking XML_Munge_git because this is a git commit
about to process
Observer has noticed that processing has begun.
finished setting up the Parser
Processing file [/var/db/ingress/message-queues/incoming/2022.07.08.20.04.11.000016.31fe9dc1198f997305ace049b055254007fc7963.xml]...
dbname = freshports.devgit
parsing file now


 *** start of all updates ***

 --- start of an update --- 

 --- end of OS --- 
OS is 'FreeBSD' : branch = 'main' for git
after converting '$Updates{branch_git}' we have 'head'
next we need to strip any leading 'branches/' prefix
OS is 'FreeBSD' : branch = 'main' for git
OS is 'FreeBSD' : branch = 'head' for git
OS is 'FreeBSD' : branch = 'head' for database names
sql = 'select SystemIDGet('FreeBSD')'
SystemBranchIDGetOrCreate has converted 'head' to 'head' which will be used in the database
sql = 'select SystemBranchIDGet(1, 'head')'
OS is 'FreeBSD' (1) : branch = head (1)
found Committer= [bofh]
found Author= [Muhammad Moinur Rahman]
OS                   = [FreeBSD]
Branch git           = [main]
branch_database_name = [head]
branch_for_files     = [head]
Committer            = [bofh]
Date                 = [2022/07/08 20:04:11 UTC]
Repository     = [doc]
Revision       = [31fe9dc1198f997305ace049b055254007fc7963]
MessageId      = [31fe9dc1198f997305ace049b055254007fc7963]
short hash     = [31fe9dc]
Subject        = [Update CORE Team liaison]
Log            = [Update CORE Team liaison

The new CORE.12 has decided on their roles about liaisons with other
administrative teams.

Approved by:	core (bofh with core-secretary@ hat on)]
into handle_message_end, let's save that message now!

xml_munge_git.pm::SaveUpdateToDB --- start
GetExistingMessageID => sql=select id from commit_log where message_id = '31fe9dc1198f997305ace049b055254007fc7963'
message 31fe9dc1198f997305ace049b055254007fc7963 has already been added to the database
no commit id returned.  we'll just exit now shall we?

Good. Now I feel better. Let’s try src next.

src

First, the magic:

[dev-ingress01 dan ~ingress/repos/src] % echo 'git tag -m "last known commit of main/origin" -f freshports/origin/main 03473e8ec8fa8d0f1ea30f85d8796ea9bf94bf29' | sudo su -fm ingress
Updated tag 'freshports/origin/main' (was ea1d45c89a1)

There we go:

*** /var/log/freshports/freshports.log ***
Jul  9 16:57:00 dev-ingress01 FreshPorts[43668]: into /usr/local/etc/periodic/everythreeminutes/100.fp_system_status
Jul  9 16:57:00 dev-ingress01 FreshPorts[43694]: into /usr/local/etc/periodic/everythreeminutes/215.fp_check_git_for_commits

*** /var/log/freshports/ingress-daemon.log ***
Jul  9 16:57:00 dev-ingress01 check_for_git_commits.sh[43698]: touching ~ingress/signals/check_git ~ingress/signals/job_waiting
Jul  9 16:57:00 dev-ingress01 check_for_git_commits.sh[43724]: done touching, going away now
Jul  9 16:57:02 dev-ingress01 ingress[71055]: yes, there is a job waiting
Jul  9 16:57:02 dev-ingress01 ingress[71055]: running /usr/local/bin/perl ./job-waiting.pl
Jul  9 16:57:02 dev-ingress01 ingress[71055]: from directory  /usr/local/libexec/freshports
Jul  9 16:57:02 dev-ingress01 ingress[71055]: -rw-r--r--  1 dan  dan  2571 Apr 24 18:40 ./job-waiting.pl

*** /var/log/freshports/freshports.log ***
Jul  9 16:57:02 dev-ingress01 FreshPorts[43818]: running job-waiting.pl 
Jul  9 16:57:02 dev-ingress01 FreshPorts[43818]: starting ./job-waiting.pl (/usr/local/libexec/freshports) 
Jul  9 16:57:02 dev-ingress01 FreshPorts[43818]: running ./job-waiting.pl as user = 'ingress' (/usr/local/libexec/freshports) 
Jul  9 16:57:02 dev-ingress01 FreshPorts[43818]: checking jobs for ingress (/usr/local/libexec/freshports) 
Jul  9 16:57:02 dev-ingress01 FreshPorts[43818]: /var/db/ingress/signals/check_git exists.  About to run check_git.sh (/usr/local/libexec/freshports) 
Jul  9 16:57:02 dev-ingress01 check_git.sh[43836]: /usr/local/libexec/freshports/check_git.sh has started

*** /var/log/freshports/git.log ***
2022.07.09 16:57:02 git-delta.sh has started. Will check these repos: 'doc ports src'
2022.07.09 16:57:02 git-delta.sh XML dir is /var/db/ingress/message-queues/incoming
2022.07.09 16:57:02 git-delta.sh Now processing repo: doc ---------------
2022.07.09 16:57:02 git-delta.sh REPODIR='/var/db/ingress/repos/doc' exists
2022.07.09 16:57:02 git-delta.sh Repodir is /var/db/ingress/repos/doc
2022.07.09 16:57:02 git-delta.sh Running: /usr/local/bin/git fetch:
2022.07.09 16:57:02 git-delta.sh fetch completed.
2022.07.09 16:57:02 git-delta.sh working on 'origin/main'
2022.07.09 16:57:02 git-delta.sh Is freshports/origin/main defined on the repo 'doc'?
2022.07.09 16:57:02 git-delta.sh running: git rev-parse -q --verify freshports/origin/main^{}
31fe9dc1198f997305ace049b055254007fc7963
2022.07.09 16:57:02 git-delta.sh the latest commit we have for freshports/origin/main is:
31fe9dc1198f997305ace049b055254007fc7963
2022.07.09 16:57:02 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/main..origin/main
2022.07.09 16:57:02 git-delta.sh Done.
2022.07.09 16:57:02 git-delta.sh No commits were found
2022.07.09 16:57:02 git-delta.sh Now processing repo: ports ---------------
2022.07.09 16:57:02 git-delta.sh REPODIR='/var/db/ingress/repos/ports' exists
2022.07.09 16:57:02 git-delta.sh Repodir is /var/db/ingress/repos/ports
2022.07.09 16:57:02 git-delta.sh Running: /usr/local/bin/git fetch:
2022.07.09 16:57:03 git-delta.sh fetch completed.
2022.07.09 16:57:03 git-delta.sh working on 'origin/2021Q2'
2022.07.09 16:57:03 git-delta.sh Is freshports/origin/2021Q2 defined on the repo 'ports'?
2022.07.09 16:57:03 git-delta.sh running: git rev-parse -q --verify freshports/origin/2021Q2^{}
d1da14bab7a800be62786aeb321b781179ea8b3f
2022.07.09 16:57:03 git-delta.sh the latest commit we have for freshports/origin/2021Q2 is:
d1da14bab7a800be62786aeb321b781179ea8b3f
2022.07.09 16:57:03 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/2021Q2..origin/2021Q2
2022.07.09 16:57:03 git-delta.sh Done.
2022.07.09 16:57:03 git-delta.sh No commits were found
2022.07.09 16:57:03 git-delta.sh working on 'origin/2021Q3'
2022.07.09 16:57:03 git-delta.sh Is freshports/origin/2021Q3 defined on the repo 'ports'?
2022.07.09 16:57:03 git-delta.sh running: git rev-parse -q --verify freshports/origin/2021Q3^{}
5d6eb9d394c16d3e3ff4816090470feb64b39239
2022.07.09 16:57:03 git-delta.sh the latest commit we have for freshports/origin/2021Q3 is:
5d6eb9d394c16d3e3ff4816090470feb64b39239
2022.07.09 16:57:03 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/2021Q3..origin/2021Q3
2022.07.09 16:57:03 git-delta.sh Done.
2022.07.09 16:57:03 git-delta.sh No commits were found
2022.07.09 16:57:03 git-delta.sh working on 'origin/main'
2022.07.09 16:57:03 git-delta.sh Is freshports/origin/main defined on the repo 'ports'?
2022.07.09 16:57:03 git-delta.sh running: git rev-parse -q --verify freshports/origin/main^{}
f46a28a1965dcaae036a25b7f387cc5304a3890a
2022.07.09 16:57:03 git-delta.sh the latest commit we have for freshports/origin/main is:
f46a28a1965dcaae036a25b7f387cc5304a3890a
2022.07.09 16:57:03 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/main..origin/main
2022.07.09 16:57:03 git-delta.sh Done.
2022.07.09 16:57:03 git-delta.sh No commits were found
2022.07.09 16:57:03 git-delta.sh Now processing repo: src ---------------
2022.07.09 16:57:03 git-delta.sh REPODIR='/var/db/ingress/repos/src' exists
2022.07.09 16:57:03 git-delta.sh Repodir is /var/db/ingress/repos/src
2022.07.09 16:57:03 git-delta.sh Running: /usr/local/bin/git fetch:
2022.07.09 16:57:03 git-delta.sh fetch completed.
2022.07.09 16:57:03 git-delta.sh working on 'origin/main'
2022.07.09 16:57:03 git-delta.sh Is freshports/origin/main defined on the repo 'src'?
2022.07.09 16:57:03 git-delta.sh running: git rev-parse -q --verify freshports/origin/main^{}
03473e8ec8fa8d0f1ea30f85d8796ea9bf94bf29
2022.07.09 16:57:03 git-delta.sh the latest commit we have for freshports/origin/main is:
03473e8ec8fa8d0f1ea30f85d8796ea9bf94bf29
2022.07.09 16:57:03 git-delta.sh Running: /usr/local/bin/git rev-list freshports/origin/main..origin/main
2022.07.09 16:57:03 git-delta.sh Done.
2022.07.09 16:57:03 git-delta.sh The commits found are:
2022.07.09 16:57:03 git-delta.sh dff31ae1c59cab9437e88bfd0f2abd35ddaa98f1
2022.07.09 16:57:03 git-delta.sh 9ef1127008ce94cf626daed346a3c1ee03063617
2022.07.09 16:57:03 git-delta.sh eec3290266bc09b4c4b4d875d2269d611adc0016
2022.07.09 16:57:03 git-delta.sh 8f9972075cb3864d47a5796eb1abdb0f4d1be8fc
2022.07.09 16:57:03 git-delta.sh aeb6948d4319fdaef8a7a6ea72717968cf5ef79b

ports

The main event:

[dev-ingress01 dan ~ingress/repos/ports] % echo 'git tag -m "last known commit of main/origin" -f freshports/origin/main db5eeb201b93c19c42724ec98f8ca1be9e88db15' | sudo su -fm ingress
Updated tag 'freshports/origin/main' (was 8f48d2bcbf66)

As I type this, the system is both creating new XML files from the ports git repo while it’s processing the above commits from the src repo. I like queues, even if it’s only implemented via files on disk in a directory. Here I am checking the queue:

[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     16:59:40
      13
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:00:07
      26
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:00:09
      37
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:00:11
     105
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:00:23
     117
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:00:25
     145
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:00:30
     156
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:00:32
     318
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:01:01
     492
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:01:30
     552
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:02:49
     552
[dev-ingress01 dan ~ingress/repos/ports] % ls ~ingress/message-queues/incoming | wc -l     17:02:51
     551
[dev-ingress01 dan ~ingress/repos/ports] %                                                 17:02:56

test

I’m collecting the above commands here for future reference, when I do stage.

cd doc
echo 'git tag -m "last known commit of main/origin" -f freshports/origin/main 21de22b400a3970b2d3d4b2831c44702698f57e7' | sudo su -fm ingress
cd -
cd src
echo 'git tag -m "last known commit of main/origin" -f freshports/origin/main 03473e8ec8fa8d0f1ea30f85d8796ea9bf94bf29' | sudo su -fm ingress
cd -
cd ports
echo 'git tag -m "last known commit of main/origin" -f freshports/origin/main db5eeb201b93c19c42724ec98f8ca1be9e88db15' | sudo su -fm ingress

I did a sudo pkg upgrade on test, then ran the above commands, then started ingress and freshports services.

All three FreshPorts nodes are now updating.

Apr 202022
 

I need some help with the git processing shell script.

Look at line 82 in this script please. See those hardcoded branch names? That’s my problem.

Let’s reduce the shell script to a simple proof of concept.

% cat ~/tmp/poc.sh
#!/bin/sh

  git for-each-ref --format '%(objecttype) %(refname)' \
      | sed -n 's/^commit refs\/remotes\///p' \
      | while read -r refname
   do
     echo "working on '$refname'"
   done

When run from the root directory of a FreeBSD ports checkout, you should see this:

[dev-ingress01 dan ~ingress/repos/ports] % ~/tmp/poc.sh
working on 'origin/2014Q1'
working on 'origin/2014Q2'
working on 'origin/2014Q3'
working on 'origin/2014Q4'
working on 'origin/2015Q1'
working on 'origin/2015Q2'
working on 'origin/2015Q3'
working on 'origin/2015Q4'
working on 'origin/2016Q1'
working on 'origin/2016Q2'
working on 'origin/2016Q3'
working on 'origin/2016Q4'
working on 'origin/2017Q1'
working on 'origin/2017Q2'
working on 'origin/2017Q3'
working on 'origin/2017Q4'
working on 'origin/2018Q1'
working on 'origin/2018Q2'
working on 'origin/2018Q3'
working on 'origin/2018Q4'
working on 'origin/2019Q1'
working on 'origin/2019Q2'
working on 'origin/2019Q3'
working on 'origin/2019Q4'
working on 'origin/2020Q1'
working on 'origin/2020Q2'
working on 'origin/2020Q3'
working on 'origin/2020Q4'
working on 'origin/2021Q1'
working on 'origin/2021Q2'
working on 'origin/2021Q3'
working on 'origin/2021Q4'
working on 'origin/2022Q1'
working on 'origin/2022Q2'
working on 'origin/HEAD'
working on 'origin/main'
[dev-ingress01 dan ~ingress/repos/ports] % 

The issue to be solved: process only those branches which use git.

The first git branch was 2021Q2 – commits prior to that branch were imported into FreshPorts via subversion.

My request to you: modify the script so that prior branches are not processed.

Using something like what you see in the code please, reproduced here:

      if [ "$refname" != "origin/2021Q2" ] && [ "$refname" != "origin/2021Q3" ] && [ "$refname" != "$MAIN_BRANCH" ]

Thank you.

EDIT 2022-04-23

Following on from Allan Jude’s suggestion to eliminate the use of sed, I liked this code:

[dev-ingress01 dan ~ingress/repos/ports] % cat ~/tmp/poc-no-sed.sh                                                                                               14:38:14
#!/bin/sh

git for-each-ref --format '%(objecttype) %(refname:lstrip=2)' refs/remotes/ --sort=-refname | 
  while read -r objtype refname
  do
#     year=${${refname#origin/}%Q*}
     quarter=${refname#origin/????Q} 
     echo "working on '$refname' - $year - $quarter"
   done
[dev-ingress01 dan ~ingress/repos/ports] % ~/tmp/poc-no-sed.sh                                                                                                   14:39:15
working on 'origin/main' -  - origin/main
working on 'origin/HEAD' -  - origin/HEAD
working on 'origin/2022Q2' -  - 2
working on 'origin/2022Q1' -  - 1
working on 'origin/2021Q4' -  - 4
working on 'origin/2021Q3' -  - 3
working on 'origin/2021Q2' -  - 2
working on 'origin/2021Q1' -  - 1
working on 'origin/2020Q4' -  - 4
working on 'origin/2020Q3' -  - 3
working on 'origin/2020Q2' -  - 2
working on 'origin/2020Q1' -  - 1
working on 'origin/2019Q4' -  - 4
working on 'origin/2019Q3' -  - 3
working on 'origin/2019Q2' -  - 2
working on 'origin/2019Q1' -  - 1
working on 'origin/2018Q4' -  - 4
working on 'origin/2018Q3' -  - 3
working on 'origin/2018Q2' -  - 2
working on 'origin/2018Q1' -  - 1
working on 'origin/2017Q4' -  - 4
working on 'origin/2017Q3' -  - 3
working on 'origin/2017Q2' -  - 2
working on 'origin/2017Q1' -  - 1
working on 'origin/2016Q4' -  - 4
working on 'origin/2016Q3' -  - 3
working on 'origin/2016Q2' -  - 2
working on 'origin/2016Q1' -  - 1
working on 'origin/2015Q4' -  - 4
working on 'origin/2015Q3' -  - 3
working on 'origin/2015Q2' -  - 2
working on 'origin/2015Q1' -  - 1
working on 'origin/2014Q4' -  - 4
working on 'origin/2014Q3' -  - 3
working on 'origin/2014Q2' -  - 2
working on 'origin/2014Q1' -  - 1

That works. Interesting how we have both main and HEAD

Jan 122022
 

One of the known limitations of mkjail is updating of jails within jails. Unless you’re running a FreshPorts node or poudriere while you’re running mkjail, this is unlikely to affect you.

For me, to avoid this stoppage, this is my plan:

  1. ssh to each jail: csshX [dev,test,stage]-ingress01
  2. disable commit processing: sudo service freshports stop
  3. stop the jail: sudo service jail stop

Then run sudo mkjail update -all on my host.

Once that is done, go back to the ssh sessions and run:

sudo mkjail update -j freshports

Start the jail:

sudo service jail start

Finally, start the freshports service:

sudo service freshports start

Yep. I worked.

Oct 232021
 

Messages such as:

Internal error: I was expecting a short description and found nothing for databases/postgresql

indicate a cache entry which predates a change to the cache format. I could clear out the entire cache, but I want to try a targeted approach. In issue 339, some detail is provided. The common thread seems to be a deleted port.

My theory: at one time these database entries contains proper data. A script which updated all ports from the repo failed to detect a deleted port and information already in the database was lost during the update.

There is a table for recording items to be cleared from cache:

freshports.devgit=# \d cache_clearing_ports
                                        Table "public.cache_clearing_ports"
   Column   |            Type             | Collation | Nullable |                     Default                      
------------+-----------------------------+-----------+----------+--------------------------------------------------
 id         | integer                     |           | not null | nextval('cache_clearing_ports_id_seq'::regclass)
 port_id    | integer                     |           | not null | 
 category   | text                        |           | not null | 
 port       | text                        |           | not null | 
 date_added | timestamp without time zone |           | not null | now()
Indexes:
    "cache_clearing_ports_pkey" PRIMARY KEY, btree (id)
    "cache_clearing_ports_port_id_idx" UNIQUE CONSTRAINT, btree (port_id)
Foreign-key constraints:
    "cache_clearing_ports_port_id_fkey" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE CASCADE ON DELETE CASCADE

This populates the table with all deleted ports:

freshports.devgit=# begin;
BEGIN
freshports.devgit=# insert into cache_clearing_ports (port_id, category, port)  
select id, (select name from categories where id = category_id), (select name from element where id = element_id) from ports where status = 'D';
INSERT 0 20715
freshports.devgit=# select count(*) from cache_clearing_ports;
 count 
-------
 20715
(1 row)

freshports.devgit=# commit;
COMMIT

This command notifies the application which is listening on the front end:

freshports.devgit=# NOTIFY port_updated;
NOTIFY

In the logs of that application, I see:

Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: Just woke up! *************
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: Got NOTIFY: pid='33939', channel='port_updated', payload=''
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: found key port_updated
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: invoking RemoveCacheEntry()
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: checking for cache entries to remove...
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: COUNT: 20715 entries to process
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: removing glob /var/db/freshports/cache/ports/databases/phpmyadmin-devel/*
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: removing glob /var/db/freshports/cache/categories/databases/*
Oct 23 12:16:33 dev-nginx01 fp-listen[69025]: DELETE FROM cache_clearing_ports WHERE id = 32289779

It eventually ends around:

Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: DELETE FROM cache_clearing_ports WHERE id = 32310493
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: finished
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: invoking ClearMiscCaches()
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: invoked: ClearMiscCaches()
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: ClearMiscCaches() is clearing out entries in /var/db/freshports/cache/news/
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: ClearMiscCaches() is removing /var/db/freshports/cache/news/index-html-head-pagesize100-pagenum1-html
Oct 23 12:31:48 dev-nginx01 fp-listen[69025]: finished: ClearMiscCaches()

That is about 15 minutes to clear out 20715 entries, roughly 1400 a minute or about 22 per second.

It is also doing one transactions per cache-clearing.

Oct 142021
 

Recently I started seeing this message in the production logs for FreshPorts:

Oct 13 15:35:38 aws-1-nginx01 FreshPorts[75290]: _pkgmessage_UCL found a type is it not prepared for : Array

What code produces that message?

GitHub makes searching for that message easy. I found it here:

syslog(LOG_ERR, '_pkgmessage_UCL found a type is it not prepared for : ' . $thing->type);

What port has this issue?

Searching the webserver logs, I find:

$ grep 15:35:38 /var/log/nginx/freshports.org-access.log.0
92.220.10.100 - - [13/Oct/2021:15:35:38 +0000] "GET /graphics/rawtherapee/?branch=2021Q3 
HTTP/1.1" 200 13068 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"

Always be able to reproduce the issue

If I cannot reproduce the issue myself, I cannot be confident that any changes I make have fixed it.

After clearing the cache for that port I was able to reproduce the error message:

$ sudo rm -rf ~freshports/cache/ports/graphics/rawtherapee

Analysis of the problem

Looking at the code I can see it is saving the pkgmessage column (from the ports table) into a file. It then converts that to JSON and parses the output to create HTML.

The code is prepared to find various action values, such as upgrade, install, and remove.

But it’s getting an array.

Duplicating the issue from the command line

Let’s look at what ucl is bringing back. This is where some command line use of psql helps. I will abbreviate the output to reduce your scrolling.

freshports.devgit=# select pkgmessage from ports_active where name = 'rawtherapee';
                               pkgmessage                                
-------------------------------------------------------------------------
 [                                                                      +
 {type: [install, upgrade], message=<<EOD                               +
 LENSFUN INFORMATION:                                                   +
                                                                        +
 This package uses lensfun to correct lens aberrations. In case         +
 your camera or lens seem unsupported, try running                      +
 lensfun-update-data - this will download new lensfun databases.        +
                                                                        +
....

 remove older cache directories.                                        +
                                                                        +
 Also, after configurations have been moved to the new version's        +
 directory, older $HOME/.config/RawTherapee* directories may be removed.+
 EOD                                                                    +
 },                                                                     +
 ]
(5 rows)

I can not use that output in a file and run it through UCL. I need to get ride of the column header (—) and the field separators (+++).

I read the output of psql –help and came up with this:

[dan@pg02:~]: $ psql --no-align --tuples-only freshports.devgit -c "select pkgmessage from ports_active where name = 'rawtherapee'"
[
{type: [install, upgrade], message=<<EOD
LENSFUN INFORMATION:

This package uses lensfun to correct lens aberrations. In case
your camera or lens seem unsupported, try running
lensfun-update-data - this will download new lensfun databases.

DISK SPACE WARNING:

...

Also, after configurations have been moved to the new version's
directory, older $HOME/.config/RawTherapee* directories may be removed.
EOD
},
]

That is much better.

Let’s feed that into a file:

[dan@pg02:~]: $ psql --no-align --tuples-only freshports.devgit -c "select pkgmessage from ports_active where name = 'rawtherapee'" > rawtherapee.ucl

Now run ucl on it (the output has been wrapped to ease reading):

[dan@pg02:~]: $ /usr/local/bin/ucl_tool --in rawtherapee.ucl --format json
[
    {
        "type": [
            "install",
            "upgrade"
        ],
        "message": "LENSFUN INFORMATION:\n\nThis package uses lensfun to correct lens 
aberrations. In case\nyour camera or lens seem unsupported, try running\nlensfun-update-data - 
this will download new lensfun databases.\n\nDISK SPACE WARNING:\n\nNote that RawTherapee uses 
version-dependent cache and configuration\ndirectories.  Please be advised that cache 
directories can grow large,\nso be sure to check all users' $HOME/.cache/RawTherapee* and have 
them\nremove older cache directories.\n\nAlso, after configurations have been moved to the new 
version's\ndirectory, older $HOME/.config/RawTherapee* directories may be removed."
    }
]

There it is. type is an array in this case. The code always assumes it is a string.

Now it’s just a matter of adjusting the code to cater for this.