In my recent post, my conclusion preferred Rename over Delete & Add. Now I’ve changed my mind.
I looked at the code, and the database. Now I prefer Delete & Add.
The database
In the database, there is no way to represent the Rename action without modifying the structure of the commit_log_elements table:
freshports.devgit=# \d commit_log_elements
Table "public.commit_log_elements"
Column | Type | Collation | Nullable | Default
---------------+--------------+-----------+----------+-------------------------------------------------
id | integer | | not null | nextval('commit_log_elements_id_seq'::regclass)
commit_log_id | integer | | not null |
element_id | integer | | not null |
revision_name | text | | |
change_type | character(1) | | not null |
The change_type field is controlled by this constraint:
Check constraints:
"commit_log_elements_change_type" CHECK (change_type = 'A'::bpchar OR change_type = 'M'::bpchar OR change_type = 'R'::bpchar OR change_type = 'r'::bpchar)
Let’s look at the comments I set up for that constraint:
freshports.devgit=# \dd commit_log_elements_change_type
Object descriptions
Schema | Name | Object | Description
--------+---------------------------------+------------------+------------------------------------------
public | commit_log_elements_change_type | table constraint | +
| | | A - add +
| | | M - modify +
| | | R - remove for subersion, delete for git+
| | | r - rename (added for git)
(1 row)
Each entry in this table documents an action (in the change_type column) and the file (element_id) upon which that action occurs. All for a given commit (commit_log_id). The revision_name column refers to the subversion revisions of a file in a given commit. For git commits, that field contains the commit hash.
Except for r (Rename). For that, we get only the origin, not the destination, of the rename.
An Add and Remove includes all the data.
A Rename does not. It omits the destination. I don’t see an easy solution for
Rename.
How much data?
We have 51 entries for r:
freshports.devgit=# select count(*) from commit_log_elements where change_type = 'r';
count
-------
51
(1 row)
We can redo all those commits easily enough.
That’s why
That explains is why I’m going with Add and Remove.
But first, let’s run a test.
The test with new XML
This is the new XML I’m going to test with:
[dan@devgit-ingress01:/var/db/ingress/message-queues/testing-new] $ cat 2020.07.27.01.58.33.000078.fd2bb768b5a39d29758b8f74bc15c321d6b4f980.xml
<?xml version='1.0' encoding='UTF-8'?>
<UPDATES Version="1.4.0.0">
<UPDATE>
<DATE Year="2020" Month="7" Day="27"/>
<TIME Timezone="UTC" Hour="1" Minute="58" Second="33"/>
<OS Repo="ports" Id="FreeBSD" Branch="master"/>
<LOG>editors/nvi2-port: Rename to editors/nvi2
Grrr... my addport notes failed me.</LOG>
<PEOPLE>
<UPDATER Handle="leres <leres@FreeBSD.org>"/>
</PEOPLE>
<COMMIT Hash="fd2bb768b5a39d29758b8f74bc15c321d6b4f980" HashShort="fd2bb76" Subject="editors/nvi2-port: Rename to editors/nvi2" EncodingLoses="false" Repository="ports"/>
<FILES>
<FILE Action="Modify" Path="editors/Makefile"/>
<FILE Action="Delete" Path="editors/nvi2-port/Makefile"/>
<FILE Action="Delete" Path="editors/nvi2-port/distinfo"/>
<FILE Action="Delete" Path="editors/nvi2-port/pkg-descr"/>
<FILE Action="Add" Path="editors/nvi2/Makefile"/>
<FILE Action="Add" Path="editors/nvi2/distinfo"/>
<FILE Action="Add" Path="editors/nvi2/pkg-descr"/>
</FILES>
</UPDATE>
</UPDATES>
I ran it through the processing:
$ sudo mv -i 2020.07.27.01.58.33.000078.fd2bb768b5a39d29758b8f74bc15c321d6b4f980.xml ~ingress/message-queues/incoming/
Looking in the logs I found this:
the size of @Files is 7 FILE ==: Modify, /ports/head/editors/Makefile, fd2bb768b5a39d29758b8f74bc15c321d6b4f980, , ports, head, editors/Makefile, 4106650 FILE ==: Delete, /ports/head/editors/nvi2-port/Makefile, fd2bb768b5a39d29758b8f74bc15c321d6b4f980, , ports, head, editors/nvi2-port/Makefile, 4106651 FILE ==: Delete, /ports/head/editors/nvi2-port/distinfo, fd2bb768b5a39d29758b8f74bc15c321d6b4f980, , ports, head, editors/nvi2-port/distinfo, 4106652 FILE ==: Delete, /ports/head/editors/nvi2-port/pkg-descr, fd2bb768b5a39d29758b8f74bc15c321d6b4f980, , ports, head, editors/nvi2-port/pkg-descr, 4106653 FILE ==: Add, /ports/head/editors/nvi2/Makefile, fd2bb768b5a39d29758b8f74bc15c321d6b4f980, , ports, head, editors/nvi2/Makefile, 4106654 FILE ==: Add, /ports/head/editors/nvi2/distinfo, fd2bb768b5a39d29758b8f74bc15c321d6b4f980, , ports, head, editors/nvi2/distinfo, 4106655 FILE ==: Add, /ports/head/editors/nvi2/pkg-descr, fd2bb768b5a39d29758b8f74bc15c321d6b4f980, , ports, head, editors/nvi2/pkg-descr, 4106656 # # # # Deleting deleted ports # # # # deleting : port = editors/nvi2-port, port_id = '59458', ' element_id = 1057410' # # # # Finished deleting deleted ports # # # #
Line 11 is the key. That’s the port being deleted this this little bit of code from verifyport.pm:
#
# $port now contains the port for this file.
# let's adjust the needs_refresh value.
#
#
# if we just deleted the Makefile for this port, there's no sense in refreshing the port.
# because it's been deleted.
#
if ($extra eq $FreshPorts::Constants::FILE_MAKEFILE && ($action eq $FreshPorts::Constants::REMOVE || $action eq $FreshPorts::Constants::DELETE)) {
#
# we are deleted (local value, never actually saved to db)
#
$port->{deleted} = 1;
print "THIS PORT HAS BEEN DELETED\n";
}
In the end, the port was properly deleted without any code changes.
Another test
Let’s also try the databases/sqliteodbc-sqlite2 commit from the previous post.
The XML is:
[dan@devgit-ingress01:/var/db/ingress/message-queues/testing-new] $ cat 2020.07.27.18.50.49.000000.c4f1589e8f8c02859c3a8893bd7a0da15b6df8b9.xml
<?xml version='1.0' encoding='UTF-8'?>
<UPDATES Version="1.4.0.0">
<UPDATE>
<DATE Year="2020" Month="7" Day="27"/>
<TIME Timezone="UTC" Hour="18" Minute="50" Second="49"/>
<OS Repo="ports" Id="FreeBSD" Branch="master"/>
<LOG>Remove expired port:
2020-07-27 databases/sqliteodbc-sqlite2: SQLite2 has stopped development in 2005, please move to SQLite3</LOG>
<PEOPLE>
<UPDATER Handle="rene <rene@FreeBSD.org>"/>
</PEOPLE>
<COMMIT Hash="c4f1589e8f8c02859c3a8893bd7a0da15b6df8b9" HashShort="c4f1589" Subject="Remove expired port:" EncodingLoses="false" Repository="ports"/>
<FILES>
<FILE Action="Modify" Path="MOVED"/>
<FILE Action="Modify" Path="databases/Makefile"/>
<FILE Action="Delete" Path="databases/sqliteodbc-sqlite2/Makefile"/>
</FILES>
</UPDATE>
</UPDATES>
The log shows:
That message is all done under Commit ID = '810547' the size of @Files is 3 FILE ==: Modify, /ports/head/MOVED, c4f1589e8f8c02859c3a8893bd7a0da15b6df8b9, , ports, head, MOVED, 4107056 FILE ==: Modify, /ports/head/databases/Makefile, c4f1589e8f8c02859c3a8893bd7a0da15b6df8b9, , ports, head, databases/Makefile, 4107057 FILE ==: Delete, /ports/head/databases/sqliteodbc-sqlite2/Makefile, c4f1589e8f8c02859c3a8893bd7a0da15b6df8b9, , ports, head, databases/sqliteodbc-sqlite2/Makefile, 4107058 # # # # Deleting deleted ports # # # # deleting : port = databases/sqliteodbc-sqlite2, port_id = '38622', ' element_id = 723652' # # # # Finished deleting deleted ports # # # #
Good!
I think we’re good to go here.
Running the old commits
Let’s get the list of old commits to run:
freshports.devgit=# select distinct revision_name from commit_log_elements where change_type = 'r';
revision_name
------------------------------------------
481991bdc9aeb780e44dae0e235b47034335375a
59a1c626e040d1eb3ef9fe05b67d0d448296f01f
75f3786fe43cd66ac10614047b0373f4d1b64696
7b86ce7d8dfa46481bb92ce3dd5cf4dc7464d9f0
8240d44f53adaf5d66bfcd4b44879cc0894d49a2
8b92b78775cfd4174e4e765d6023b195fee0bbfa
a2810a57b46b7e367c0b1aabb18a03f9d5971be6
d016c9140a6e4b7627d26485926a659f2686185d
d38f3fa0fc45e73bdb17bbc139d846f784f3d1e8
d5c503dd5c31a156c7cc5b78afbf98cb2643d716
dbbc6bd62c9b904b7237df4344f55cdc2bbcce1f
e792298a3b48d74277e293cc864a113958a39bd1
e9f48a1168c84f84424d95816af645895e338daf
f2bfe60090b840b6d99a3288c0b745843cefcfe1
(14 rows)
That’s an easy list to run.
Here’s my script to delete the existing commits:
freshports.devgit=# begin; BEGIN freshports.devgit=# freshports.devgit=# delete from commit_log where svn_revision in freshports.devgit-# ( freshports.devgit(# '481991bdc9aeb780e44dae0e235b47034335375a', freshports.devgit(# '59a1c626e040d1eb3ef9fe05b67d0d448296f01f', freshports.devgit(# '75f3786fe43cd66ac10614047b0373f4d1b64696', freshports.devgit(# '7b86ce7d8dfa46481bb92ce3dd5cf4dc7464d9f0', freshports.devgit(# '8240d44f53adaf5d66bfcd4b44879cc0894d49a2', freshports.devgit(# '8b92b78775cfd4174e4e765d6023b195fee0bbfa', freshports.devgit(# 'a2810a57b46b7e367c0b1aabb18a03f9d5971be6', freshports.devgit(# 'd016c9140a6e4b7627d26485926a659f2686185d', freshports.devgit(# 'd38f3fa0fc45e73bdb17bbc139d846f784f3d1e8', freshports.devgit(# 'd5c503dd5c31a156c7cc5b78afbf98cb2643d716', freshports.devgit(# 'dbbc6bd62c9b904b7237df4344f55cdc2bbcce1f', freshports.devgit(# 'e792298a3b48d74277e293cc864a113958a39bd1', freshports.devgit(# 'e9f48a1168c84f84424d95816af645895e338daf', freshports.devgit(# 'f2bfe60090b840b6d99a3288c0b745843cefcfe1'); DELETE 14 freshports.devgit=# commit; COMMIT freshports.devgit=#
#!/bin/sh commits="481991bdc9aeb780e44dae0e235b47034335375a 59a1c626e040d1eb3ef9fe05b67d0d448296f01f 75f3786fe43cd66ac10614047b0373f4d1b64696 7b86ce7d8dfa46481bb92ce3dd5cf4dc7464d9f0 8240d44f53adaf5d66bfcd4b44879cc0894d49a2 8b92b78775cfd4174e4e765d6023b195fee0bbfa a2810a57b46b7e367c0b1aabb18a03f9d5971be6 d016c9140a6e4b7627d26485926a659f2686185d d38f3fa0fc45e73bdb17bbc139d846f784f3d1e8 d5c503dd5c31a156c7cc5b78afbf98cb2643d716 dbbc6bd62c9b904b7237df4344f55cdc2bbcce1f e792298a3b48d74277e293cc864a113958a39bd1 e9f48a1168c84f84424d95816af645895e338daf f2bfe60090b840b6d99a3288c0b745843cefcfe1" for commit in $commits do echo /usr/local/libexec/freshports/test-new-xml-code-single.sh ports $commit | sudo su -fm ingress done
That will create the new and old versions of the XML in two different directories. I take the new stuff and toss it into the incoming queue.
[dan@devgit-ingress01:/var/db/ingress/message-queues/testing-new] $ ls -l total 195 -rw-rw-r-- 1 ingress ingress 887 Jul 30 22:03 2019.09.18.17.37.59.000000.f2bfe60090b840b6d99a3288c0b745843cefcfe1.xml -rw-rw-r-- 1 ingress ingress 1291 Jul 30 22:03 2020.07.11.18.08.50.000000.e792298a3b48d74277e293cc864a113958a39bd1.xml -rw-rw-r-- 1 ingress ingress 1246 Jul 30 22:02 2020.07.11.20.45.27.000000.59a1c626e040d1eb3ef9fe05b67d0d448296f01f.xml -rw-rw-r-- 1 ingress ingress 1316 Jul 30 22:02 2020.07.15.15.47.41.000000.7b86ce7d8dfa46481bb92ce3dd5cf4dc7464d9f0.xml -rw-rw-r-- 1 ingress ingress 2492 Jul 30 22:03 2020.07.17.08.21.24.000000.d38f3fa0fc45e73bdb17bbc139d846f784f3d1e8.xml -rw-rw-r-- 1 ingress ingress 2492 Jul 30 22:02 2020.07.17.08.21.54.000000.481991bdc9aeb780e44dae0e235b47034335375a.xml -rw-rw-r-- 1 ingress ingress 7161 Jul 30 22:03 2020.07.21.17.15.38.000000.d016c9140a6e4b7627d26485926a659f2686185d.xml -rw-rw-r-- 1 ingress ingress 1316 Jul 30 22:02 2020.07.22.10.41.53.000000.8b92b78775cfd4174e4e765d6023b195fee0bbfa.xml -rw-rw-r-- 1 ingress ingress 982 Jul 30 22:03 2020.07.22.14.00.59.000000.e9f48a1168c84f84424d95816af645895e338daf.xml -rw-rw-r-- 1 ingress ingress 82325 Jul 30 22:02 2020.07.23.16.31.18.000000.75f3786fe43cd66ac10614047b0373f4d1b64696.xml -rw-rw-r-- 1 ingress ingress 908 Jul 30 22:03 2020.07.26.08.01.07.000000.d5c503dd5c31a156c7cc5b78afbf98cb2643d716.xml -rw-rw-r-- 1 ingress ingress 1168 Jul 30 22:03 2020.07.28.18.52.11.000000.a2810a57b46b7e367c0b1aabb18a03f9d5971be6.xml -rw-rw-r-- 1 ingress ingress 1490 Jul 30 22:03 2020.07.30.12.20.24.000000.dbbc6bd62c9b904b7237df4344f55cdc2bbcce1f.xml -rw-rw-r-- 1 ingress ingress 37100 Jul 30 22:02 2020.07.30.20.39.39.000000.8240d44f53adaf5d66bfcd4b44879cc0894d49a2.xml [dan@devgit-ingress01:/var/db/ingress/message-queues/testing-new] $ sudo mv * ~ingress/message-queues/incoming/ [dan@devgit-ingress01:/var/db/ingress/message-queues/testing-new] $
After waiting a bit, I ran this query again:
freshports.devgit=# select distinct revision_name from commit_log_elements where change_type = 'r';
revision_name
------------------------------------------
f2bfe60090b840b6d99a3288c0b745843cefcfe1
(1 row)
freshports.devgit=#
So why didn’t that one run. Let’s see the XML:
<?xml version='1.0' encoding='UTF-8'?>
<UPDATES Version="1.4.0.0">
<UPDATE>
<DATE Year="2019" Month="9" Day="18"/>
<TIME Timezone="UTC" Hour="17" Minute="37" Second="59"/>
<OS Repo="ports" Id="FreeBSD" Branch="master"/>
<LOG>net-mgmt/unifi5: Update to 5.11.46
Also pull in port improvements from glewis@ to enable customizing which Java
that Unifi runs with
PR: 240016</LOG>
<PEOPLE>
<UPDATER Handle="feld <feld@FreeBSD.org>"/>
</PEOPLE>
<COMMIT Hash="f2bfe60090b840b6d99a3288c0b745843cefcfe1" HashShort="f2bfe60" Subject="net-mgmt/unifi5: Update to 5.11.46" EncodingLoses="false" Repository="ports"/>
<FILES>
<FILE Action="Modify" Path="net-mgmt/unifi5/Makefile"/>
<FILE Action="Modify" Path="net-mgmt/unifi5/distinfo"/>
<FILE Action="Modify" Path="net-mgmt/unifi5/files/unifi.in"/>
</FILES>
</UPDATE>
</UPDATES>
No renames. What’s up with that? Only modifies. Let’s check the database.
I found a lot of them…
freshports.devgit=# select count(*) from commit_log_elements where revision_name = 'f2bfe60090b840b6d99a3288c0b745843cefcfe1'; count ------- 3189 (1 row)
Over a lot of commits:
freshports.devgit=# select count(distinct commit_log_id) from commit_log_elements where revision_name = 'f2bfe60090b840b6d99a3288c0b745843cefcfe1'; count ------- 813 (1 row)
This brings to mind a hardcode commit_id I recall in the code….
Searching my blog, I found subversion vs git – in the database.
Searching the code, I found:
$ grep -r f2bfe60090b840b6d99a3288c0b745843cefcfe1 ~/scripts/ ~/modules/
modules/xml_munge_git.pm:# $Updates{commit_hash} = 'f2bfe60090b840b6d99a3288c0b745843cefcfe1';
So I did have it hardcoded for a while. That explains it. I can ignore that commit.
I think this merge can be accepted with some slight changes.
