This post talks about vuxml entries I found in the FreshPorts database table which were not in the security/vuxml files. This is another post in the new processing of vuxml series:
- How FreshPorts processes vuxml entries
- The new process_vuxml.pl
- Why does this PostgreSQL \copy fail when launched by the daemon?
When the new code was completed, it migrated on to other nodes. That is where the missing vuxml entries became apparent. The situation most likely existed on that original node but was undetected because the development process wound up deleting all vuxml entries instead of only the intended entries.
Symptoms
I first noticed the issue when a message like this came in:
Sep 4 02:51:05 dev-ingress01 process_vuxml.sh[33165]: vuxml starts Sep 4 02:51:27 dev-ingress01 process_vuxml_import_temp_file.py[35247]: Starting up Sep 4 02:51:27 dev-ingress01 process_vuxml_import_temp_file.py[35247]: copying in from /tmp/freshports_vuxml_processing_OwJs6 Sep 4 02:51:27 dev-ingress01 process_vuxml_import_temp_file.py[35247]: copying completed. 6018 rows copied. Sep 4 02:51:27 dev-ingress01 process_vuxml_import_temp_file.py[35247]: Finishing Sep 4 02:51:29 dev-ingress01 FreshPorts[33220]: FATAL: we were deleting way too many rows from vuxml: 71 items, which is more than our expected minimum value of 10 (/usr/local/libexec/freshports)
That FATAL message was a sanity check added in when errors in importing led to all vuxml entries being deleted. At first, I thought this was an error for me to fix. Later, I realized it was missing data.
The FreeBSD Forum Post
I created a post on the FreeBSD Forums. Eventually I discovered the commit which added one of the missing vulns (picked for no particular reason).
Through a not-quite-a-proper-binary search, I found the commit which removed the data.
“Purge another batch of superceded www/chromium entries to give us additional headroom under the 5M vuln.xml file size limit” : 2017-09-29
It was intentional. Way back when, one file contained all the vulns. Today, it is split up by year. There was a need to keep the file under 5MB in size. This commit removed a bunch of old vulns.
Tips for this type of search
First, I obtained a list of commits for security/vuxml:
[21:11 mydev dvl /usr/ports] % git log — security/vuxml > ~/tmp/security-vuxml.log
I used that list of commits and dates to checkout the ports tree as of that commit. I did that checkout like this:
[21:09 mydev dvl /usr/ports] % sudo git checkout f50795c81a7d5280d1c358ee571f71ffa94153df — security/vuxml
That greatly reduced the time it took to get the files I wanted to check.
date did not work
When checking out by date, I received warnings such as this. That is why I went for the commit checkouts instead.
[11:36 mydev dvl /usr/ports] % sudo git checkout 'main@{2014-05-01 18:30:00}' -- security/vuxml
warning: log for 'main' only goes back to Tue, 27 Jul 2021 17:10:04 +0000
warning: log for 'main' only goes back to Tue, 27 Jul 2021 17:10:04 +0000
I suspect that is the date the FreeBSD ports repository moved over to git from subversion.
Should we add the deleted vulns back in?
For completeness, yes. For effectiveness, no. Those vulns were removed in 2017 – if you’re still running software from 2017, that’s impressive.
Here are the vulns in question:
freshports.test=# SELECT V.vid, V.checksum, V.date_entry
FROM vuxml V
LEFT OUTER JOIN vuxml_import vi
ON V.vid = vi.vid
WHERE vi.vid IS NULL order by V.date_entry
;
vid | checksum | date_entry
--------------------------------------+------------------------------------------------------------------+------------
68ac6266-25c3-11e1-b63a-00262d5ed8ee | b8e087fe8e92682b0108e542a2be540887a774d030838f2336fd369c9805691e | 2011-12-13
1a1aef8e-3894-11e1-8b5c-00262d5ed8ee | 92ca8e5ee91bb462638c21aabf39e684f44d4d4839bd641ae4baaae92cd40ee1 | 2012-01-06
33d73d59-4677-11e1-88cd-00262d5ed8ee | 0a90f6430de7ac2941f0e002b11894209f4a034660b99bbe0e6f47b77ba4c7d0 | 2012-01-24
fe1976c2-5317-11e1-9e99-00262d5ed8ee | a0e84c565229219375e6bf7f50ab061a5c09ded9ed0f166708d79fce3cd24350 | 2012-02-09
2f5ff968-5829-11e1-8288-00262d5ed8ee | 0c1f803e069b09afc8c9fe074d6640a12334685ea8869a85d13e3922ac56e4ed | 2012-02-15
99aef698-66ed-11e1-8288-00262d5ed8ee | 42ec0dbb14eabc1ff247069077792fc2a85e6a3c002d930ab4682b5a7125f3c4 | 2012-03-05
1015e1fe-69ce-11e1-8288-00262d5ed8ee | 026f56318faf27b65c525f54a3827d57e79e88e28a33232951f7822e100a80e6 | 2012-03-09
ab1f515d-6b69-11e1-8288-00262d5ed8ee | 99196a85bd3b7fb7a8da92bb87b144598f7c9eeadee73f6d3a40b238704e6b40 | 2012-03-11
330106da-7406-11e1-a1d7-00262d5ed8ee | 9a73285e151f6222d3abab84dbcef1332c2f8957dd861ba4ba2e6d6f8774bec0 | 2012-03-22
b8f0a391-7910-11e1-8a43-00262d5ed8ee | aded3cedf2a8bb3221c03dcfed9428bbd72da045b66cd35a7e158161a3befdfc | 2012-03-28
057130e6-7f61-11e1-8a43-00262d5ed8ee | 74806756c5f02faa5df0c1a1932fbfaaafa6bda0f82137520335bb15309cb31e | 2012-04-05
94c0ac4f-9388-11e1-b242-00262d5ed8ee | 3ff24b4c7374365e7b698ad9a6d5838ee2e77e5af5437c3ccaa1bb43824027b2 | 2012-05-01
1449af37-9eba-11e1-b9c1-00262d5ed8ee | 0698fe1d1d6a18480156c13400049c164350ad41e837480beaedf51827846898 | 2012-05-15
219d0bfd-a915-11e1-b519-00262d5ed8ee | e218157b166b49fadf5c46f48fa1d6a87872727039562866c604bf273a0d1be7 | 2012-05-28
ff922811-c096-11e1-b0f4-00262d5ed8ee | 42d394b4750e1ce5ac70e7a7979733e650227540478fdb84d93ff62a297e7111 | 2012-06-27
10f38033-e006-11e1-9304-000000000000 | 7a974d74cf780e41081d64bdc60d4b9256949e1f4f2d542cd2f637b1c4068847 | 2012-08-06
60bbe12c-e2c1-11e1-a8ca-00262d5ed8ee | 902e97e7e018e88fab18f0ca5a1029b55859b47fca823f7ec1166093ac9554d0 | 2012-08-10
ce84e136-e2f6-11e1-a8ca-00262d5ed8ee | b4bd5bec153ce9b577686b9599d769949df4f2ef8f7b6b6151b624a4096c11cf | 2012-08-10
2092a45b-e2f6-11e1-a8ca-00262d5ed8ee | f4c69a5be9592025302f515f48b40c8111e32b2abb3d1127aef615167b043cd9 | 2012-08-10
ee68923d-f2f5-11e1-8014-00262d5ed8ee | a524e86e5296b1703cb5cafbade0a2f0670f9e4e3927609e6726fbb00f399848 | 2012-08-30
5bae2ab4-0820-11e2-be5f-00262d5ed8ee | ab6820c3b1ef7d8410903f64545a46c0f1e5cf2b7173adaec0c60c6fbf9edf0b | 2012-09-26
e6161b65-1187-11e2-afe3-00262d5ed8ee | dba3305c21832bcc5cfb53691ec89a3afac05a490a579a70eeaa494e8f754002 | 2012-10-08
09e83f7f-1326-11e2-afe3-00262d5ed8ee | 030e26975da2dfab17917c032e201671679eb3ddd27988fd4afa9085d145972e | 2012-10-10
209c068d-28be-11e2-9160-00262d5ed8ee | be440b58c3723d194c36bfde0be8488a9f068fcae5dc9a20b81fad0f9cdd2dc1 | 2012-11-07
4d64fc61-3878-11e2-a4eb-00262d5ed8ee | 759db2cf258999aa3eba87d1439d53932cdda6dcd8196fa6752cbc3ab570a459 | 2012-11-27
5af51ae9-3acd-11e2-a4eb-00262d5ed8ee | c953014e38431912ff29904baf9d53ba7dcdd5043b20f7ffd6159ef23984071c | 2012-11-30
51f84e28-444e-11e2-8306-00262d5ed8ee | 997da1b3b1d6e049280e378e7f0f92538747fd0963ca7c85e10645180371748c | 2012-12-12
46bd747b-5b84-11e2-b06d-00262d5ed8ee | b30b2e51052f7c40a9eafd54dc590ccb2b0e6bf420ceed8d25ef54071daa006e | 2013-01-11
8d03202c-6559-11e2-a389-00262d5ed8ee | 669cb34da2666f01e44cc22715f66613e412c0d2df7d380b8fd58b81e60320c3 | 2013-01-23
dfd92cb2-7d48-11e2-ad48-00262d5ed8ee | c234d358aed2facdeef68f6fece9bb36da774ceb914d49612719610fd9ed039d | 2013-02-22
40d5ab37-85f2-11e2-b528-00262d5ed8ee | d3feead907b061c0c56bdeec0bc7b6bd89b34c82a4d104fb8ac4806ff781019b | 2013-03-06
54bed676-87ce-11e2-b528-00262d5ed8ee | 8916103c809d25c96bccb516bcd2959d384d89aa79cf85f0de2e01b6af3f39ee | 2013-03-08
bdd48858-9656-11e2-a9a8-00262d5ed8ee | 90fadd57aa0cb5653dc16ee061279a849f7bbfbf30086b9d5e7c14295a8b51c9 | 2013-03-26
358133b5-c2b9-11e2-a738-00262d5ed8ee | 967c2bb7a2eba3e1a1e8433200d51e396a372845dfccd39e8d4d004913610784 | 2013-05-22
4865d189-cd62-11e2-ae11-00262d5ed8ee | 8d77a8511b5b34f31fd878648f17266d4f254eb4a41e9fed759495b7ad85b383 | 2013-06-04
3b80104f-e96c-11e2-8bac-00262d5ed8ee | da960a3d6569bd55b1e1b9a2d2dd73917b083d0e4be809a1d296c051ed872e09 | 2013-07-10
69098c5c-fc4b-11e2-8ad0-00262d5ed8ee | e996b5a10fd627286a245d9501737883aa1c21fd6730cb621af21818f3489952 | 2013-08-03
ae651a4b-0a42-11e3-ba52-00262d5ed8ee | 1aa0e40348098893e463270e7a4a3bb6637ef1b7f9ed50aa123561aafbfef8d5 | 2013-08-21
e5414d0c-2ade-11e3-821d-00262d5ed8ee | 0941757d6a266b27ac0e12588c8a50a66c05278f8a3cde157ebab7f3407bfa53 | 2013-10-01
710cd5d5-35cb-11e3-85f9-00262d5ed8ee | 5e1e18e989c98be6ab4e8aa3d260276eac5c3d8d02bc9e8c3983d8b19ba58c20 | 2013-10-15
3bfc7016-4bcc-11e3-b0cf-00262d5ed8ee | 125cf8fa6e99bb1f83e793bbb02b147564346acfae0d8005ffbd687bb00f7446 | 2013-11-12
e62ab2af-4df4-11e3-b0cf-00262d5ed8ee | a4c9230949257665c17eeaa89c34d5e8fd72b2999b91ce476638275d5e6c4a6e | 2013-11-15
79356040-5da4-11e3-829e-00262d5ed8ee | 8d8ed6299c7ee373c5c515b829cf54b586eb0a176207faf933c9a6a5b8eabd93 | 2013-12-05
5acf4638-7e2c-11e3-9fba-00262d5ed8ee | c7944e999ba0e63d49ae673d7729f1f03ac0882c04460ebcb841490ae0b6766b | 2014-01-15
f9810c43-87a5-11e3-9214-00262d5ed8ee | 1f833f36bd543089e876047dc4c13388c1330c764fe688b7f0c9b961e1b98fc4 | 2014-01-27
9dd47fa3-9d53-11e3-b20f-00262d5ed8ee | 059823598075bbb964e92b81bfcd7e1e5ea4dedba2650acbd7e5264fa49c9df8 | 2014-02-24
b4023753-a4ba-11e3-bec2-00262d5ed8ee | dc442c4ffde05828f0a0d8a979aaa2207f87e89578f31b356bf131c910cb30e3 | 2014-03-05
24cefa4b-a940-11e3-91f2-00262d5ed8ee | e3446a2bc3d805cf16499751b029f54b9a5a9209a52d7a8c9fcee76d76e1ca48 | 2014-03-11
a70966a1-ac22-11e3-8d04-00262d5ed8ee | 3397dfc593d221f69561be4164e6d483145689cd284b8b51ad493d4de1c9c81f | 2014-03-15
963413a5-bf50-11e3-a2d6-00262d5ed8ee | 202055879052a9b6631228c7a31dc90024d699edd8da5d6bf174d789f046235f | 2014-04-08
7cf25a0c-d031-11e3-947b-00262d5ed8ee | 6034e91827f284f41d3564594618e8cc09d51ced3ba1ced09671f63338ebdc35 | 2014-04-30
cdf450fc-db52-11e3-a9fc-00262d5ed8ee | 89156b8e90065a6c1ffdef6b84ee5fc9a8bcc6405f661af9b31de76abe19c93e | 2014-05-14
64f3872b-e05d-11e3-9dd4-00262d5ed8ee | 5b76a06fcbb2dc6ccaf32f0087a50eea5aafd4e62db516bbc5072d9f2baaa12f | 2014-05-20
0b0fb9b0-f0fb-11e3-9bcd-000c6e25e3e9 | 6383ffbf3bb450666c3bab15b917943f1baedca1a7c79d6a65a19aa754bae362 | 2014-06-10
3718833e-0d27-11e4-89db-000c6e25e3e9 | eed5e9843e33c9a905406ca192fb79d00a78beb2278e0c667f36fce29b93192e | 2014-07-16
df7754c0-2294-11e4-b505-000c6e25e3e9 | c16d592d3d31ff26952ddca9a60e54390c6737b645bb567b3494e4fef18e28b8 | 2014-08-13
fd5f305d-2d3d-11e4-aa3d-00262d5ed8ee | 5cb4040c24155b07b8fdb79af6ea4165e232a658c0c16bc8e8f4349fdf001710 | 2014-08-26
36a415c8-3867-11e4-b522-00262d5ed8ee | 028f4906a46265d81279811ce459605c7d3aee740ccb0036af3b38fe3d8c3024 | 2014-09-09
bd2ef267-4485-11e4-b0b7-00262d5ed8ee | 448a23106bd3ae1d621d568e86f028502979aae7560f95c6f213da7889f68a42 | 2014-09-25
d2bbcc01-4ec3-11e4-ab3f-00262d5ed8ee | 97779c383d9b2ab8ce13012280750d1ce382bee2da9329ad49568fa50ac6a824 | 2014-10-08
734bcd49-aae6-11e4-a0c1-c485083ca99c | a89843ffe34f477edab9ab6fd18729f9634a1413af607a96785f349d02fe5b06 | 2015-02-02
6bedc863-9fbe-11e8-945f-206a8a720317 | d439e649c97834769a127b300871fbdc02d7cfad82ee1b7f23cd4aca71ec4565 | 2018-08-14
e3404a6e-4364-11ea-b643-206a8a720317 | 22fdb2b3cb4e40a42139b458fef53c80f11a61a0dcf45ecb2105207bbe8c12dc | 2020-01-30
40c75597-574a-11ea-bff8-c85b76ce9b5a | 9afe665a589f7d75acf1bcea50b2f16467be59e342ec30d88ff228b582588a64 | 2020-02-24
76f1ce19-5749-11ea-bff8-c85b76ce9b5a | 993bf9691c6915fa26a4a1e6ca966904e7cff637cebac1f374df45a26de94009 | 2020-02-24
4a8a-8987-11ea-93ef-b42e99a1b9c3 | cd50e51ae64c274ecc3a08317bcb6250a1c6de6586590cace39a9b8cc1791b33 | 2020-06-24
3749ae9e-f132-11ea-97da-d05099c0ae8c | e5ca4fa7e1d942384704546b4f4b5e76d66ddb169f7c31d9da0ea3e7aedbe889 | 2020-09-07
bbda3d16-968e-11ee-b780-b42e991fc52e | e6571c89a21eef3e80d58951a8412f3ff0e8bfdfd84b122c082d03a75ca8451e | 2023-12-09
46a9eb0f-d7d2-11ee-bb12-001b217b3468 | 63c2b51152ddc1533daf3b846dd3263cdc3b087912d3eaf224b3d4444e35dc40 | 2024-03-01
44de1b82-662d-11ef-a51b-b42e991fc52e | d3be39200cdd30e2e1450ccece270e2c703038587086d2615e27fd2ad41750d5 | 2024-08-29
7b4ffa5b-5bc5-11f0-834f-b42e991fc52e | b6e618f5fda6830f1b0138474dbc84c9501b2e74489bb43346a5c74adeba4b95 | 2025-07-08
(71 rows)
freshports.test=#
