Setting up a FreeBSD ports git repo for FreshPorts usage

I’m documenting this as a go along because I’m sure I’ll be doing something that I later wish I had written down.

Create the repo

This creates a FreeBSD git-based port repo.

echo git clone https://github.com/freebsd/freebsd-ports.git /var/db/freshports/ports-jail/var/db/repos/PORTS-head-git | sudo su -l freshports

The latest local commit

This tells us the hash of the latest commit contained within this local copy of the repo.

Let’s call this the base commit.

$ cd PORTS-head-git
[dan@devgit-ingress01:/var/db/freshports/ports-jail/var/db/repos/PORTS-head-git] $ git log -n 1
commit 240cab957e5c13c50ed539ed7bc7d511bb1c3368 (HEAD -> master, origin/master, origin/HEAD)
Author: yuri 
Date:   Tue Jun 30 16:10:10 2020 +0000

    math/fplll: Update 5.3.2 -> 5.3.3
    
    Reported by:    portscout

Now, we do this as the freshports user.

Pull the latest details:

[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ git pull
warning: Pulling without specifying how to reconcile divergent branches is
discouraged. You can squelch this message by running one of the following
commands sometime before your next pull:

  git config pull.rebase false  # merge (the default strategy)
  git config pull.rebase true   # rebase
  git config pull.ff only       # fast-forward only

You can replace "git config" with "git config --global" to set a default
preference for all repositories. You can also pass --rebase, --no-rebase,
or --ff-only on the command line to override the configured default per
invocation.

Already up to date.
[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ 

Let’s try that recommended command:

[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ git config pull.rebase false
[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ git pull
Already up to date.

Wait a bit longer for more commits

Some time later:

[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ git pull
remote: Enumerating objects: 345, done.
remote: Counting objects: 100% (308/308), done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 150 (delta 94), reused 137 (delta 81), pack-reused 0
Receiving objects: 100% (150/150), 30.17 KiB | 9.00 KiB/s, done.
Resolving deltas: 100% (94/94), completed with 70 local objects.
From https://github.com/freebsd/freebsd-ports
   240cab957e5c..008e30ab98fb  master          -> origin/master
   6c5a4e63b897..f412caa04aa1  branches/2020Q2 -> origin/branches/2020Q2
   3d3db8513a0a..a70ca3962b88  svn_head        -> origin/svn_head
Updating 240cab957e5c..008e30ab98fb
Fast-forward
 audio/jamulus/Makefile                                        |    2 +-
 audio/jamulus/distinfo                                        |    6 +-
 biology/bamtools/Makefile                                     |    2 +-
 databases/py-redis/Makefile                                   |    2 +-
 databases/py-redis/distinfo                                   |    6 +-
 devel/cargo-c/Makefile                                        |   68 +--
 devel/cargo-c/distinfo                                        |  138 +++---
 devel/cmake-gui/Makefile                                      |    1 +
 devel/cmake/Makefile                                          |    1 +
 devel/opendht/Makefile                                        |    2 +-
 graphics/open3d/Makefile                                      |    2 +-
 graphics/ossim/Makefile                                       |    2 +-
 lang/solidity/Makefile                                        |    1 +
 lang/swi-pl/Makefile                                          |   76 +--
 lang/swi-pl/distinfo                                          |    6 +-
 lang/swi-pl/files/extra-patch-skip-gnu-qsort                  |   55 ---
 lang/swi-pl/files/patch-packages_clib_cmake_FindLibUUID.cmake |   14 +
 lang/swi-pl/pkg-plist                                         | 3677 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------------------------------------------------
 math/cadabra2/Makefile                                        |    1 +
 math/libqalculate/Makefile                                    |    6 +-
 math/libqalculate/distinfo                                    |    8 +-
 math/libqalculate/pkg-plist                                   |    2 +-
 math/py-pynndescent/Makefile                                  |    2 +-
 math/py-pynndescent/distinfo                                  |    6 +-
 math/py-uncertainties/Makefile                                |    2 +-
 math/py-uncertainties/distinfo                                |    6 +-
 math/qalculate/Makefile                                       |    2 +-
 math/qalculate/distinfo                                       |    6 +-
 math/qalculate/pkg-plist                                      |   15 +-
 misc/ezc3d/Makefile                                           |    2 +-
 misc/ezc3d/distinfo                                           |    6 +-
 misc/ignition-fuel-tools/Makefile                             |    2 +-
 multimedia/libopenshot/Makefile                               |    1 +
 net-p2p/eiskaltdcpp-daemon/Makefile                           |    2 +-
 net/coturn/Makefile                                           |    2 +-
 net/coturn/distinfo                                           |    6 +-
 net/flower/Makefile                                           |    2 +-
 net/libjson-rpc-cpp/Makefile                                  |    2 +-
 science/py-tensorflow/Makefile                                |    2 +-
 sysutils/fusefs-securefs/Makefile                             |    1 +
 www/chromium/Makefile                                         |    1 +
 www/domoticz/Makefile                                         |    1 +
 www/iridium/Makefile                                          |    2 +-
 www/qt5-webengine/Makefile                                    |    1 +
 x11-themes/qtcurve-kf5/Makefile                               |    2 +-
 x11-themes/qtcurve-qt5/Makefile                               |    2 +-
 x11-themes/qtcurve/Makefile                                   |   23 +-
 x11-themes/qtcurve/distinfo                                   |    8 +-
 x11-themes/qtcurve/pkg-plist.kde4                             |   21 -
 x11-themes/qtcurve/pkg-plist.utils                            |   27 ++
 x11/polybar/Makefile                                          |    1 +
 x11/waybar/Makefile                                           |    2 +-
 52 files changed, 2148 insertions(+), 2088 deletions(-)
 delete mode 100644 lang/swi-pl/files/extra-patch-skip-gnu-qsort
 create mode 100644 lang/swi-pl/files/patch-packages_clib_cmake_FindLibUUID.cmake
 delete mode 100644 x11-themes/qtcurve/pkg-plist.kde4
[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ 

OK, now we have more commits!

List all the commits since the base commit

After a git clone https://github.com/FreshPorts/git_proc_commit I modified git_proc_commit/git-show-commit.sh:

[dan@devgit-ingress01:~/src/git_proc_commit] $ git diff
diff --git a/git-show-commit.sh b/git-show-commit.sh
index fb20d30..4385933 100755
--- a/git-show-commit.sh
+++ b/git-show-commit.sh
@@ -1,14 +1,14 @@
 #!/bin/sh
 
 # the latest commit we have processed
-latest="3bd153c2494182bb89915e6fc9222288c154285f" # [NEW PORT]: devel/py-oci - Wed Sep 18 17:14:34 2019 +0000
+latest="240cab957e5c13c50ed539ed7bc7d511bb1c3368" # [NEW PORT]: devel/py-oci - Wed Sep 18 17:14:34 2019 +0000
 
 # before running this script:
 # mkdir ~/src
 # cd ~/src
 # git clone git@github.com:freebsd/freebsd-ports.git
 
-repo_dir=~/src/freebsd-ports
+repo_dir=~freshports/ports-jail/var/db/repos/PORTS-head-git
 
 GIT="/usr/local/bin/git"
 
[dan@devgit-ingress01:~/src/git_proc_commit] $ 

Basically, the path and the commit hash were updated. That hash I added in was obtained via the git log -n 1 command.

When running it:

[freshports@devgit-ingress01 ~/ports-jail/var/db/repos/PORTS-head-git]$ ~dan/src/git_proc_commit/git-show-commit.sh
008e30ab98fba636554ada737157baa84debedb3
46560d7e66714e5d574c69e42ab408f3181c1614
1c7120178a876f6daa217abd64cf8b715de46bd8
12ad02023e15ffeaeca60c78c64c11f9736bbac0
f6a1f3e3c8e17a09dbffefd8debcf3c44861e713
99d2fba63577a91850c9b311d0028c461a4a0b29
42855a0b23b363889207c7d0de5f122982bce37d
21e3f0999b283d1a440c7d0d078b325b138874f2
4046ad10a608fdcb2d6bd71a8e5d4361ad0dc069
4983ea53d2c3b0dfc377c87d6529ca18c76ce229
6f21f15631a6b851cc2f1fc8b2e59c64d808e94f
164aa7374784c77ea68d79875ab3c79950802f4b

Those are the commits which occurred since the previous git pull.

With this information, we can iterate over the commits, processing them one at a time.

First script

This was used for cvs commits, and then svn commits. Now it’s for git commits. This is untested. I’m out of time.

[dan@devgit-ingress01:~/scripts] $ svn di
Index: freebsd-git.sh
===================================================================
--- freebsd-git.sh	(revision 5375)
+++ freebsd-git.sh	(working copy)
@@ -1,25 +1,25 @@
 #!/bin/sh
 #
-# $Id: freebsd-cvs.sh,v 1.9 2011-08-15 16:31:56 dan Exp $
+# $Id: freebsd-git.sh,v 1.9 2011-08-15 16:31:56 dan Exp $
 #
 # Copyright (c) 1999-2003 DVL Software
 #
-# Process a raw mail message by converting it to XML, then importing it into
+# Process a raw git log  message by converting it to XML, then importing it into
 # the database.
 #
-# Takes a file name as a parameter
+# Takes a git commit hash a parameter
 #
-LOGGERTAG="freebsd-cvs.sh"
+LOGGERTAG="freebsd-git.sh"
 
 if [ $# -ne 1 ]
 then
-   echo $0 : usage $0 FILE
+   echo $0 : usage $0 COMMIT_HASH
    exit 1
 fi
 
 if [ ! -f config.sh ]
 then
-	echo "config.sh not found by freebsd-cvs.sh..."
+	echo "config.sh not found by freebsd-git.sh..."
 	exit 1
 fi
 
@@ -37,18 +37,23 @@
 
 ${LOGGER} -t ${LOGGERTAG} "$0 invoked, using XML='${XML}' and OUTPUT='${OUTPUT}'"
 
-PATHNAME=$1
+COMMIT_HASH=$1
 
-FILE=`basename ${PATHNAME}` 
+PROCESS_ID=${$}
+FILE=`date +%Y.%m.%d.%H.%M.%S`.$PROCESS_ID.${COMMIT_HASH}.txt
 
 #
 # convert the raw file to XML
 #
-${LOGGER} -t ${LOGGERTAG} "$0 converting to XML via process_mail.pl"
-${LOGGER} -t ${LOGGERTAG} /usr/local/bin/perl ${SCRIPTDIR}/process_mail.pl from ${PATHNAME} into ${XML}/${FILE}.xml errors to ${XML}/${FILE}.errors
+${LOGGER} -t ${LOGGERTAG} "$0 converting to XML via git-to-freshports-xml.py"
 
-/usr/local/bin/perl ${SCRIPTDIR}/process_mail.pl < ${PATHNAME} >    \
-       ${XML}/${FILE}.xml 2>${XML}/${FILE}.errors
+echo 
+# output file 1to commit_hash.process_id so that if we process the same hash again, it does not conflict
+${LOGGER} -t ${LOGGERTAG} ${SCRIPTDIR}/git-to-freshports-xml.py --path ${FRESHPORTS_JAIL_BASE_DIR}${PORTSDIR} --commit ${COMMIT_HASH} --output ${XML}/${FILE}.xml
+
+exit
+
+${SCRIPTDIR}/git-to-freshports-xml.py --path ${FRESHPORTS_JAIL_BASE_DIR}${PORTSDIR} --commit ${COMMIT_HASH} --output ${XML}/${FILE}.xml 2>${XML}/${FILE}.errors
 RESULT=$?
 
 if [ -f ${XML}/${FILE}.errors ]
@@ -68,8 +73,7 @@
 
 ${LOGGER} -t ${LOGGERTAG} "$0 loading that XML into the database via load_xml_into_db.pl"
 
-/usr/local/bin/perl ${SCRIPTDIR}/load_xml_into_db.pl ${XML}/${FILE}.xml > \
-               ${OUTPUT}/${FILE}.loading 2>${OUTPUT}/$FILE.errors
+/usr/local/bin/perl ${SCRIPTDIR}/load_xml_into_db.pl ${XML}/${FILE}.xml ${OUTPUT}/${FILE}.loading 2>${OUTPUT}/$FILE.errors
 RESULT=$?
 
 if [ -f ${OUTPUT}/$FILE.errors ]

What’s next

I need to alter the script which invokes freebsd-git.sh so that it passes in the git commit hash.

Something like this, with $i set to something obtained from git-show-commit.sh as shown above.

[dan@devgit-ingress01:~/src/fp-daemon] $ svn di
Index: fp-daemon.sh
===================================================================
--- fp-daemon.sh	(revision 5375)
+++ fp-daemon.sh	(working copy)
@@ -92,7 +92,7 @@
 				else
 					${LOGGER} -t ${LOGGERTAG} "processing ${i}"
 
-					/bin/sh ./freebsd-cvs.sh ${i}
+					/bin/sh ./freebsd-git.sh ${i}
 
 					RESULT=$?
 					${LOGGER} -t ${LOGGERTAG} "result=$RESULT"
[dan@devgit-ingress01:~/src/fp-daemon] $ 

Small steps.