The doc repo has moved from svn to git. This changeover occurred on 2020-12-09.
The last svn commit was: 54737
The first git commit was: 3be01a475855e7511ad755b2defd2e0da5d58bbe
To date, devgit.freshports.org has been using https://github.com/freebsd/freebsd-doc/ for processing commits.
Today’s work will convert from that GitHub repo to https://cgit.freebsd.org/doc/ (actually, https://git.freebsd.org/).
What changes are required
The following changes are required:
- A new working copy of the git.FreeBSD.org/doc repo
- A marker pointing the last commit processed
- Configuration file changes to point to that repo
EDIT: 2021-01-01 – NOTE: all the configuration files are also maintained via Ansible – the manual steps are not necessarily required. I think I should next document the changes to Ansible when changing database names / servers.
The rest of the post documents those changes.
A new working copy of the git.FreeBSD.org/doc repo
Let’s clone that repo:
$ cd /var/db/ingress/repos [ingress@devgit-ingress01 ~]$ cd repos [ingress@devgit-ingress01 ~/repos]$ ls -l total 18 drwxr-xr-x 26 ingress ingress 45 Dec 17 03:03 freebsd drwxr-xr-x 23 ingress ingress 26 Dec 8 07:12 freebsd-doc´ drwxr-xr-x 69 ingress ingress 84 Dec 17 02:18 freebsd-ports drwxr-xr-x 69 ingress ingress 84 Nov 22 19:28 freebsd-ports-quarterly -rw-r--r-- 1 ingress ingress 41 Dec 17 23:51 latest.freebsd -rw-r--r-- 1 ingress ingress 41 Dec 17 23:51 latest.freebsd-doc -rw-r--r-- 1 ingress ingress 41 Dec 17 23:51 latest.freebsd-ports -rw-r--r-- 1 ingress ingress 41 Dec 17 23:51 latest.freebsd-ports-quarterly [ingress@devgit-ingress01 ~/repos]$ git clone https://git.FreeBSD.org/doc.git Cloning into 'doc'... remote: Enumerating objects: 449334, done. remote: Counting objects: 100% (449334/449334), done. remote: Compressing objects: 100% (120607/120607), done. remote: Total 449334 (delta 314272), reused 448476 (delta 313566), pack-reused 0 Receiving objects: 100% (449334/449334), 245.50 MiB | 11.60 MiB/s, done. Resolving deltas: 100% (314272/314272), done. Updating files: 100% (11346/11346), done. [ingress@devgit-ingress01 ~/repos]$
Done.
A marker pointing the last commit processed
In the directory listing above, you can see files starting with latest. which mirror the list of repos, which are directories. We need to create a new file, named latest.doc, which contains the hash of the last commit processed by FreshPorts.
To find that value, I did this:
- browse to https://cgit.freebsd.org/doc/
- click on main
- scroll down to “Mark the repository as being converted to Git.”
- Take that hash value
You won’t be able to reproduce that once there are enough commits and that particular commit scrolls off the page. You will always be able to find the first commit (3be01a475855e7511ad755b2defd2e0da5d58bbe) and see parent listed as 89d0233560e4ba181d73143fc25248b407120e09
Let’s put that into a file:
[ingress@devgit-ingress01 ~/repos]$ echo 89d0233560e4ba181d73143fc25248b407120e09 > latest.doc
What do we have now:
[ingress@devgit-ingress01 ~/repos]$ ls -l total 21 drwxr-xr-x 23 ingress ingress 27 Dec 17 23:55 doc drwxr-xr-x 26 ingress ingress 45 Dec 17 03:03 freebsd drwxr-xr-x 23 ingress ingress 26 Dec 8 07:12 freebsd-doc drwxr-xr-x 69 ingress ingress 84 Dec 17 02:18 freebsd-ports drwxr-xr-x 69 ingress ingress 84 Nov 22 19:28 freebsd-ports-quarterly -rw-r--r-- 1 ingress ingress 41 Dec 18 00:06 latest.doc -rw-r--r-- 1 ingress ingress 41 Dec 18 00:06 latest.freebsd -rw-r--r-- 1 ingress ingress 41 Dec 18 00:06 latest.freebsd-doc -rw-r--r-- 1 ingress ingress 41 Dec 18 00:06 latest.freebsd-ports -rw-r--r-- 1 ingress ingress 41 Dec 18 00:06 latest.freebsd-ports-quarterly [ingress@devgit-ingress01 ~/repos]$
Next, configuration file changes.
Configuration file changes to point to that repo
What files need changing? Just one I think.
[dan@devgit-ingress01:~] $ cd /usr/local/etc/freshports [dan@devgit-ingress01:/usr/local/etc/freshports] $ sudo grep freebsd-doc * config.sh: doc) dir='freebsd-doc';;
When I make this change, I want to disable the git processing so as not to change a configuration setting in the middle of processing.
$ sudo sysrc -f /etc/periodic.conf fp_check_for_git_commits_enable="NO" fp_check_for_git_commits_enable: YES -> NO
Let’s verify nothing is running git processing:
ps auwwx | grep ingress ingress 60714 0.0 0.0 10676 2188 - SCJ 00:14 0:00.00 sleep 3 ingress_svn 60727 0.0 0.0 10676 2188 - SCJ 00:14 0:00.00 sleep 3 ingress_svn 98912 0.0 0.0 11004 2424 - IsJ Sat23 0:00.15 daemon: ingress_svn[98913] (daemon) ingress_svn 98913 0.0 0.0 11868 2968 - SJ Sat23 0:32.78 /bin/sh /usr/local/libexec/freshports-service/ingress_svn.sh ingress 98921 0.0 0.0 11004 2424 - IsJ Sat23 0:00.44 daemon: ingress[98923] (daemon) ingress 98923 0.0 0.0 11844 2952 - SJ Sat23 0:21.70 /bin/sh /usr/local/libexec/freshports-service/ingress.sh dan 60729 0.0 0.0 11384 2764 2 S+J 00:14 0:00.00 grep ingress
That’s normal. ingress_svn is the daemon checking for incoming svn commits.
ingress is the daemon which looks for incoming git commits, but that’s the daemon which processes the XML files. The periodic.conf setting we adjusted is for the creation of those XML files. That is what we are pausing now.
Next, I updated config.sh. This is what we have now:
[dan@devgit-ingress01:/usr/local/etc/freshports] $ sudo grep doc * config.pm:# Values as found at https://www.postgresql.org/docs/current/static/libpq-ssl.html config.pm:$FreshPorts::Config::Repo_DOC = 'doc'; config.pm:$FreshPorts::Config::DB_Root_Prefix_DOC = '/doc'; config.sh:# see https://www.postgresql.org/docs/12/libpq-envars.html config.sh: doc) dir='doc';;
That last line is the updated valud. The others refer to repo names and internal pathnames within the FreshPorts database.
What’s next?
Turn on git commit processing and watch the logs.
$ sudo sysrc -f /etc/periodic.conf fp_check_for_git_commits_enable="YES" fp_check_for_git_commits_enable: NO -> YES $
Now I wait until 7:24, because the script runs every 3 minutes.
BOOM! The latest commit is now in: 482d8311b8a1e25a66ee49af4bc7efadd8be22aa
