Latest Vulnerabilities … aren’t

May 6th, 2012

Ruslan wrote in to mention that the ‘Latest Vulnerabilities’ section is not current. I’m trying to find out why.

FreshPorts has processed the latest vuln entries; http://www.freshports.org/vuxml.php reports Revision: 1.2685.

But http://www.freshports.org/vuxml.php?all lists nothing later than 2012-04-27.

I just refreshed the security/vuxml port on the main FreshPorts server. It is now re-processing the latest vuln file. Let’s see what that gives me.

Ummm. nothing different. OK, let’s delete all existing vuxml entries from the table and try again:

$ touch ../dynamic/vuxml ../dynamic/job_waiting && tail -F /var/log/messages
May  6 15:03:27 supernews fp-daemon: yes, there is a job waiting
May  6 15:03:28 supernews FreshPorts[12281]: flag not set.  no work for process_updating.sh
May  6 15:03:28 supernews FreshPorts[12281]: flag not set.  no work for process_moved.sh
May  6 15:03:28 supernews FreshPorts[12281]: flag not set.  no work for process_www_en_ports_categories.sh
May  6 15:03:28 supernews FreshPorts[12281]: /usr/websites/freshports.org/dynamic/vuxml exists.  About to run process_vuxml.sh
May  6 15:03:28 supernews FreshPorts /usr/websites/freshports.org/scripts/process_vuxml.sh: vuxml processing begins
May  6 15:03:29 supernews FreshPorts /usr/websites/freshports.org/scripts/process_vuxml.sh: vuxml ident begins
May  6 15:03:29 supernews FreshPorts /usr/websites/freshports.org/scripts/process_vuxml.sh: vuxml latest begins
May  6 15:03:30 supernews FreshPorts /usr/websites/freshports.org/scripts/process_vuxml.sh: vuxml finishes
May  6 15:03:30 supernews FreshPorts /usr/websites/freshports.org/scripts/process_vuxml.sh: vuxml terminates
May  6 15:03:30 supernews FreshPorts[12281]: Finished running process_vuxml.sh
May  6 15:03:31 supernews sudo:   nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/libexec/nagios/check_pid_sudo /var/run/dovecot/master.pid

Umm, that’s a problem. vuxml should take many minutes to run.

Running the command manually shows an error

[dan@supernews:/usr/websites/freshports.org/scripts] $ perl ./process_vuxml.pl -w <  ~dan/ports/security/vuxml/vuln.xml
Existing VuXML entries will be deleted
dbname = freshports.org
creating parsing engine
running parsing engine
vid:           60de13d5-95f0-11e1-806a-001143cd36d8
topic:         php -- vulnerability in certain CGI-based setups
packages:
    php5:
               gt: 5.4       lt: 5.4.2
               lt: 5.3.12        php53:
               lt: 5.3.12        php4:
               lt: 4.4.10        php52:
               lt: 5.2.17_8  description:

	<p>php development team reports:</p>
	<blockquote cite="http://www.php.net/archive/2012.php#id2012-05-03-1">
	  <p>Security Enhancements and Fixes in PHP 5.3.12:</p>
	  <ul>
	    <li>Initial fix for cgi-bin ?-s cmdarg parse issue
	      (CVE-2012-1823)</li>
	  </ul>
	</blockquote>

references:
    cvename:   CVE-2012-1823
dates:
    discovery: 2012-05-03
    entry:     2012-05-05

-----------------------------
sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252497,
				2145201,
				'package')
gt: 5.4       lt: 5.4.2
lt: 5.3.12    sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252498,
				2145201,
				'package')
lt: 5.3.12    sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252499,
				2145201,
				'package')
lt: 4.4.10    sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252500,
				2145201,
				'package')
lt: 5.2.17_8      cvename:   CVE-2012-1823
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817623,
				2145201,
				'cvename',
				'CVE-2012-1823')
vid:           18dffa02-946a-11e1-be9d-000c29cc39d3
topic:         WebCalendar -- multiple vulnerabilities
packages:
    WebCalendar-devel:
               le: 1.2.4     description:

	<p>Hanno Boeck reports:</p>
	<blockquote cite="http://www.openwall.com/lists/oss-security/2012/04/28/1">
	  <p>Fixes [are now available] for various security vulnerabilities
            including LFI (local file inclusion), XSS (cross site scripting)
            and others.</p>
	</blockquote>

references:
    cvename:   CVE-2012-1495
    cvename:   CVE-2012-1496
    url:       http://packetstormsecurity.org/files/112332/WebCalendar-1.2.4-Remote-Code-Execution.html
    url:       http://packetstormsecurity.org/files/112323/WebCalendar-1.2.4-Pre-Auth-Remote-Code-Injection.html
    url:       http://archives.neohapsis.com/archives/bugtraq/2012-04/0182.html
dates:
    discovery: 2012-04-28
    entry:     2012-05-02

-----------------------------
sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252501,
				2145202,
				'package')
le: 1.2.4         cvename:   CVE-2012-1495
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817624,
				2145202,
				'cvename',
				'CVE-2012-1495')
    cvename:   CVE-2012-1496
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817625,
				2145202,
				'cvename',
				'CVE-2012-1496')
    url:       http://packetstormsecurity.org/files/112332/WebCalendar-1.2.4-Remote-Code-Execution.html
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817626,
				2145202,
				'url',
				'http://packetstormsecurity.org/files/112332/WebCalendar-1.2.4-Remote-Code-Execution.html')
    url:       http://packetstormsecurity.org/files/112323/WebCalendar-1.2.4-Pre-Auth-Remote-Code-Injection.html
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817627,
				2145202,
				'url',
				'http://packetstormsecurity.org/files/112323/WebCalendar-1.2.4-Pre-Auth-Remote-Code-Injection.html')
    url:       http://archives.neohapsis.com/archives/bugtraq/2012-04/0182.html
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817628,
				2145202,
				'url',
				'http://archives.neohapsis.com/archives/bugtraq/2012-04/0182.html')
vid:           94c0ac4f-9388-11e1-b242-00262d5ed8ee
topic:         chromium -- multiple vulnerabilities
packages:
    chromium:
               lt: 18.0.1025.168description:

	<p>Google Chrome Releases reports:</p>
	<blockquote cite="http://googlechromereleases.blogspot.com/search/label/Stable%20updates">
	  <p>[106413] High CVE-2011-3078: Use after free in floats handling.
	    Credit to Google Chrome Security Team (Marty Barbella) and
	    independent later discovery by miaubiz.</p>
	  <p>[117627] Medium CVE-2011-3079: IPC validation failure. Credit to
	    PinkiePie.</p>
	  <p>[121726] Medium CVE-2011-3080: Race condition in sandbox IPC.
	    Credit to Willem Pinckaers of Matasano.</p>
	  <p>[121899] High CVE-2011-3081: Use after free in floats handling.
	    Credit to miaubiz.</p>
	  <p>[117110] High CVE-2012-1521: Use after free in xml parser. Credit
	    to Google Chrome Security Team (SkyLined) and independent later
	    discovery by wushi of team509 reported through iDefense VCP
	    (V-874rcfpq7z).</p>
	</blockquote>

references:
    cvename:   CVE-2011-3078
    cvename:   CVE-2011-3079
    cvename:   CVE-2011-3080
    cvename:   CVE-2011-3081
    cvename:   CVE-2012-1521
    url:       http://googlechromereleases.blogspot.com/search/label/Stable%20updates
dates:
    discovery: 2012-04-30
    entry:     2012-05-01

-----------------------------
sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252502,
				2145203,
				'package')
lt: 18.0.1025.168    cvename:   CVE-2011-3078
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817629,
				2145203,
				'cvename',
				'CVE-2011-3078')
    cvename:   CVE-2011-3079
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817630,
				2145203,
				'cvename',
				'CVE-2011-3079')
    cvename:   CVE-2011-3080
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817631,
				2145203,
				'cvename',
				'CVE-2011-3080')
    cvename:   CVE-2011-3081
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817632,
				2145203,
				'cvename',
				'CVE-2011-3081')
    cvename:   CVE-2012-1521
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817633,
				2145203,
				'cvename',
				'CVE-2012-1521')
    url:       http://googlechromereleases.blogspot.com/search/label/Stable%20updates
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817634,
				2145203,
				'url',
				'http://googlechromereleases.blogspot.com/search/label/Stable%20updates')
vid:           2cde1892-913e-11e1-b44c-001fd0af1a4c
topic:         php -- multiple vulnerabilities
packages:
    php53:
               lt: 5.3.11        php5:
               lt: 5.3.11    description:

          <p>php development team reports:</p>
          <blockquote cite="http://www.php.net/archive/2012.php#id2012-04-26-1">
	    <p>Security Enhancements for both PHP 5.3.11 and PHP 5.4.1:</p>
	    <ul>
	      <li>Insufficient validating of upload name leading to corrupted $_FILES indices. (CVE-2012-1172) </li>
	      <li>Add open_basedir checks to readline_write_history and readline_read_history.</li>
	    </ul>
	    <p>Security Enhancements for both PHP 5.3.11 only:</p>
	    <ul>
	      <li>Regression in magic_quotes_gpc fix for CVE-2012-0831.</li>
	    </ul>
          </blockquote>

references:
    cvename:   CVE-2012-0831
    cvename:   CVE-2012-1172
    url:       http://www.php.net/archive/2012.php#id2012-04-26-1
dates:
    discovery: 2012-03-01
    entry:     2012-04-28
    modified:  2012-05-04

-----------------------------
sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252503,
				2145204,
				'package')
lt: 5.3.11    sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252504,
				2145204,
				'package')
lt: 5.3.11        cvename:   CVE-2012-0831
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817635,
				2145204,
				'cvename',
				'CVE-2012-0831')
    cvename:   CVE-2012-1172
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817636,
				2145204,
				'cvename',
				'CVE-2012-1172')
    url:       http://www.php.net/archive/2012.php#id2012-04-26-1
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817637,
				2145204,
				'url',
				'http://www.php.net/archive/2012.php#id2012-04-26-1')
vid:           0fa15e08-92ec-11e1-a94a-00215c6a37bb
topic:         samba -- incorrect permission checks vulnerability
packages:
    samba34:
               gt: 3.4.*     lt: 3.4.17
    samba35:
               gt: 3.5.*     lt: 3.5.15
    samba36:
               gt: 3.6.*     lt: 3.6.5
description:

	<p>The Samba project reports:</p>
	<blockquote cite="http://www.samba.org/samba/security/CVE-2012-2111">
	  <p>Samba versions 3.4.x to 3.6.4 inclusive are affected
	    by a vulnerability that allows arbitrary users to modify
	    privileges on a file server.</p>
	  <p>Security checks were incorrectly applied to the Local
	    Security Authority (LSA) remote proceedure calls (RPC)
	    CreateAccount, OpenAccount, AddAccountRights and
	    RemoveAccountRights allowing any authenticated user
	    to modify the privileges database.</p>
	  <p>This is a serious error, as it means that authenticated
	    users can connect to the LSA and grant themselves the
	    "take ownership" privilege. This privilege is used by the
	    smbd file server to grant the ability to change ownership
	    of a file or directory which means users could take ownership
	    of files or directories they do not own.</p>
	</blockquote>

references:
    cvename:   CVE-2012-2111
dates:
    discovery: 2012-04-30
    entry:     2012-04-30

-----------------------------
sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252505,
				2145205,
				'package')
gt: 3.4.*     lt: 3.4.17
sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252506,
				2145205,
				'package')
gt: 3.5.*     lt: 3.5.15
sql is insert into vuxml_affected(id, vuxml_id, type) values (
				3252507,
				2145205,
				'package')
gt: 3.6.*     lt: 3.6.5
    cvename:   CVE-2012-2111
sql is insert into vuxml_references(id, vuxml_id, type, reference) values (
				7817638,
				2145205,
				'cvename',
				'CVE-2012-2111')
references_push(): Missing reference value at /usr/local/lib/perl5/site_perl/5.8.9/mach/XML/Parser/Expat.pm line 469
[dan@supernews:/usr/websites/freshports.org/scripts] $

The problem is caused by the vid entry after the above.

b428e6b3-926c-11e1-8d7b-003067b2972c contains this around line 276 of vuln.xml:

    <references>
      <freebsdsa/>
    </references>

If I delete that freebsdsa line, all goes well.

This is valid XML. However, the code expects something in there.

The template in the code for this section is:

    >vuxml>vuln>references
    >vuxml>vuln>references>url *
    >vuxml>vuln>references>mlist *
    >vuxml>vuln>references>cvename *
    >vuxml>vuln>references>bid *
    >vuxml>vuln>references>certsa *
    >vuxml>vuln>references>certvu *
    >vuxml>vuln>references>uscertsa *
    >vuxml>vuln>references>uscertta *
    >vuxml>vuln>references>freebsdsa *
    >vuxml>vuln>references>freebsdpr *

vuln.xml has been updated to give this tag some content. Ideally, my code should work without. At another time of year, I could look into this deeper, but… BSDCan.

But…

This appears to be the fix:

1331c1331
<     $VuXML->references_push( FREEBSDSA, $VuXML->{text_buffer} );
---
>     $VuXML->references_push( FREEBSDSA, $VuXML->{text_buffer} // '' );

FYI, this code was last changed 7 years, 3 months ago. My thanks to Mr Matthew Seaman for his fine work on the vuxml processing.

Posted in Development | No Comments »

ports-mgmt/pkg not found

March 31st, 2012

Baptiste Daroussin (bapt) brought my attention to a new port today. It’s about three months old, but it wasn’t showing up in FreshPorts. If you look at ports-mgmt/pkg now, it should look like this:

Bad port. Bad bad port.

bapt speculated that perhaps pkg was a reserved word in FreshPorts. I thought not. I was wrong. It is a special case in FreshPorts. But it took a while to find that out.

First, we checked dev and beta to see if they were having the same issue. They were.

Next, let’s look at the database:

freshports.org=# select * from element where name = 'pkg' and parent_id = (select id from element where name = 'ports-mgmt');
   id   | name | parent_id | directory_file_flag | status
--------+------+-----------+---------------------+--------
 405152 | pkg  |    265343 | D                   | A
(1 row)
freshports.org=#

OK, pkg is definitely in the database as a directory. Good.

Does pkg have any children?

freshports.org=# select * from element where parent_id = 405152;
   id   |    name     | parent_id | directory_file_flag | status
--------+-------------+-----------+---------------------+--------
 413289 | files       |    405152 | D                   | A
 413282 | pkg-plist   |    405152 | F                   | A
 406674 | pkg-message |    405152 | F                   | A
 405155 | pkg-descr   |    405152 | F                   | A
 405154 | distinfo    |    405152 | F                   | A
 405153 | Makefile    |    405152 | F                   | A
(6 rows)

freshports.org=#

Yes, that looks right. Good.

I started looking through the web code to see if there was a problem there. I found one. The database does not know this is a port:

freshports.org=# select * from elementGet('/ports/ports-mgmt/pkg');
   id   | name | type | status | iscategory | isport |   element_pathname
--------+------+------+--------+------------+--------+-----------------------
 405152 | pkg  | D    | A      | f          | f      | /ports/ports-mgmt/pkg
(1 row)

freshports.org=#

See isport? It should be true.

FreshPorts logs everyone commit it processes. Going to the logs for the initial commit for this port, I find:

STARTING _CompileListOfPorts ................................
FILE ==: Add, ports/Mk/bsd.pkgng.mk, 1.1, ports, Mk, bsd.pkgng.mk, 1785995
YES, this file is in the ports tree
... but is on the list of IgnoredItems!

FILE ==: Modify, ports/Mk/bsd.port.mk, 1.704, ports, Mk, bsd.port.mk, 1785996
YES, this file is in the ports tree
... but is on the list of IgnoredItems!

FILE ==: Modify, ports/ports-mgmt/Makefile, 1.49, ports, ports-mgmt, Makefile, 1785997
YES, this file is in the ports tree
... but is on the list of IgnoredItems!

FILE ==: Add, ports/ports-mgmt/pkg/Makefile, 1.1, ports, ports-mgmt, pkg, Makefile, 1785998
YES, this file is in the ports tree
... but is on the list of IgnoredItems!

FILE ==: Add, ports/ports-mgmt/pkg/distinfo, 1.1, ports, ports-mgmt, pkg, distinfo, 1785999
YES, this file is in the ports tree
... but is on the list of IgnoredItems!

FILE ==: Add, ports/ports-mgmt/pkg/pkg-descr, 1.1, ports, ports-mgmt, pkg, pkg-descr, 1786000
YES, this file is in the ports tree
... but is on the list of IgnoredItems!

ENDING _CompileListOfPorts ................................

IgnoredItems? OK, that sounds promising. Looking in scripts/constants.pm I find:

#
# These are the entries within /usr/ports/ which we ignore
# and /usr/ports/ which FreshPorts does not track
#
%FreshPorts::Constants::IgnoredItems = (
        “Attic”                 => 1,
        “distfiles”             => 2,
        “Mk”                    => 3,
        “Tools”                 => 4,
        “Templates”             => 5,
        “Makefile”              => 6,
        “pkg”                   => 7,
        “Makefile.inc”  => 8,
);

For fun, I removed pkg, and renumberd Makefile.inc to 7. Then I reran the initial commit. Success. We have a port. Then I reran all other commits for the port. Everything was caught up.

However, it’s niggling at me… wondering how IgnoredItems is used and why it caught pkg.

Looking in scripts/verifyport.pm I see:

# is this file is in the ports tree?
# e.g. ports/LEGAL won't get through here because $port_name will not be defined.
if ($subtree eq $FreshPorts::Config::ports_prefix && defined($category_name) && defined($port_name)) {
    print "YES, this file is in the ports tree\n";

    if (!defined($FreshPorts::Constants::IgnoredItems{$category_name}) && !defined($FreshPorts::Constants::IgnoredItems{$port_name})) {
        # find the port for this filename....
...
    } else {
        print "... but is on the list of IgnoredItems!\n\n";
    }
} else {
    print "that file isn't in the ports tree\n";
}

It sounds like I can be a bit more specific with my tests. Looking for each value in IgnoredPorts, I find these items:

dan@kraken:~] $ find /usr/ports/ -name Attic
[dan@kraken:~] $ find /usr/ports/ -name distfiles
/usr/ports/distfiles
[dan@kraken:~] $ find /usr/ports/ -name Mk
/usr/ports/Mk
[dan@kraken:~] $ find /usr/ports/ -name Tools
/usr/ports/Tools
[dan@kraken:~] $ find /usr/ports/ -name Templates
/usr/ports/Templates
[dan@kraken:~] $
[dan@kraken:~] $ find /usr/ports/ -name Makefile.inc
/usr/ports/arabic/Makefile.inc
/usr/ports/audio/xmms-faad/Makefile.inc
/usr/ports/chinese/Makefile.inc
/usr/ports/devel/p4/Makefile.inc
/usr/ports/devel/ecore-main/Makefile.inc
/usr/ports/devel/eric4/Makefile.inc
/usr/ports/devel/subversion16/Makefile.inc
/usr/ports/editors/xxe/Makefile.inc
/usr/ports/french/Makefile.inc
/usr/ports/games/anki/Makefile.inc
/usr/ports/german/Makefile.inc
/usr/ports/graphics/evas-core/Makefile.inc
/usr/ports/hebrew/Makefile.inc
/usr/ports/hungarian/Makefile.inc
/usr/ports/japanese/Makefile.inc
/usr/ports/korean/Makefile.inc
/usr/ports/misc/gnome2-reference/Makefile.inc
/usr/ports/net-im/licq/Makefile.inc
/usr/ports/net/ntp-rc/Makefile.inc
/usr/ports/net/ntp-devel/Makefile.inc
/usr/ports/net/pcnfsd/files/Makefile.inc
/usr/ports/net/ntp/Makefile.inc
/usr/ports/net/samba-libsmbclient/Makefile.inc
/usr/ports/news/husky-base-devel/Makefile.inc
/usr/ports/news/husky-base/Makefile.inc
/usr/ports/polish/Makefile.inc
/usr/ports/ports-mgmt/pkg_install/files/Makefile.inc
/usr/ports/portuguese/Makefile.inc
/usr/ports/russian/Makefile.inc
/usr/ports/textproc/aspell/Makefile.inc
/usr/ports/ukrainian/Makefile.inc
/usr/ports/vietnamese/Makefile.inc
/usr/ports/x11-themes/irssi-themes/Makefile.inc
[dan@kraken:~] $

Not very many items. In effect, I want to look for, and ignore these items:

  1. ports/distfiles
  2. ports/Mk
  3. ports/Tools
  4. ports/Tempaltes

It may be better to have a list of ignored ‘categories’ (e.g. Mk). Not that Mk is a category…

Posted in Bug fixes | 7 Comments »

Mk/bsd.perl.mk bug causes 5000 errrors….

March 10th, 2012

Well, perhaps not 5000 errors, but quite a lot of errors.

A recent change to Mk/bsd.perl.mk has caused all sorts of grief on two of the FreshPorts servers. The problem occurs when specifying PORTDIR. Here is an example:

#!/bin/sh

make -V BUILD_DEPENDS -V FORBIDDEN \
-f /usr/home/dan/ports/databases/p5-DBIx-Class/Makefile DISTDIR=/usr/ports/distfiles PORTSDIR=/usr/home/dan/ports

make -V BUILD_DEPENDS -V FORBIDDEN \
-f /usr/home/dan/ports/databases/p5-DBIx-Class/Makefile DISTDIR=/usr/ports/distfiles

The difference in the output for perl.

/usr/local/bin/perl5.8.9:/usr/home/dan/ports/lang/
/usr/local/bin/perl5.8.9:/usr/ports/lang/perl5.8

That is the problem. There is no port called lang. Thus, when FreshPorts tries to add the dependencies, it errors.

The system which works is using revision 1.22 of perl.mk. The system which fails is using revision 1.23.

The diff is:

 diff bsd.perl.mk ~dan/ports/Mk/bsd.perl.mk | less
4c4
< # $FreeBSD: ports/Mk/bsd.perl.mk,v 1.23 2012/03/08 18:37:53 pgollucci Exp $
---
> # $FreeBSD: ports/Mk/bsd.perl.mk,v 1.22 2011/08/15 06:50:00 sunpoet Exp $
70a71,74
> .elif  ${PERL_LEVEL} >= 501000
> PERL_PORT?=   perl5.10
> .else # ${PERL_LEVEL} < 501000
> PERL_PORT?=   perl5.8

Before you ask, this is the setting on all systems, regardless of success or failure:

$ grep PERL_VERSION /etc/make.conf
PERL_VERSION=5.8.9

The fix is as follows:

$ diff  ~dan/ports/Mk/bsd.perl.mk ~dan/ports/Mk/bsd.perl.mk~
71,72d70
< .else
< PERL_PORT?=     perl5.8

Thanks to aDe and melflynn for helping me track this down and come up with a fix to stop the errors.

Hopefully this will be fixed in Mk/ soon.

Posted in Bug fixes | 3 Comments »

VUXML - doing it faster

January 23rd, 2012

It takes a long time for FreshPorts to process a new vuln.xml file. A very long time. Why? Because it process all entries in the file. Every time.

Anton Berezin proposed an idea. I paraphrase, but in brief, he said:

Figuring out what changed is as easy as hashing an entry and storing it in a db

That’s a very good start. I’ve checked the code, and the sticking point is going to be is table:

freshports.org=# \d ports_vulnerable
    Table "public.ports_vulnerable"
 Column  |  Type   |     Modifiers
---------+---------+--------------------
 port_id | integer | not null
 current | integer | not null default 1
 past    | integer | not null default 0
Indexes:
    "ports_vulnerable_pkey" PRIMARY KEY, btree (port_id)
Foreign-key constraints:
    "$1" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE RESTRICT ON DELETE CASCADE
Triggers:
    ports_vulnerable_delete_clear_cache AFTER DELETE ON ports_vulnerable FOR EACH ROW EXECUTE PROCEDURE ports_vulnerable_delete_clear_cache()

freshports.org=#

This table is used to decide whether or not to display a grey or black skull at the top of each port page. This table is populated with two different queries. The first, figures out the number of vulnerabilities, current and past, that affect a given port:

  SELECT count(distinct vuxml_id)
    INTO l_VulnCount
    FROM commit_log_ports_vuxml
   WHERE port_id = p_PortID;

The second, determines the number of vulnerabilities that currently affect the port:

    SELECT count(distinct vuxml_id)
      INTO l_VulnCurrent
      FROM commit_log_ports_vuxml CLPV, commit_log_ports CLP, ports P, ports_vulnerable PV
     WHERE CLP.commit_log_id = CLPV.commit_log_id
       AND CLPV.port_id      = CLP.port_id
       AND P.id              = CLP.port_id
       AND P.version         = CLP.port_version
       AND P.revision        = CLP.port_revision
       AND CLP.port_epoch    = P.portepoch
       AND PV.port_id        = P.id
       AND P.id              = p_PortID
     GROUP BY P.id;

NOTE: I am not sure of the reason for referencing ports_vulnerable here… My first thought is it is unnecessary. I’m not sure how it is to reference a table that is just about to get updated.

The difference between the two values, if any, is the number of past vulnerabilities.

The problem is an edge case where a vuln entry affects a port, then does not affect a port. For example, consider that a new vuln entry is created and mistakenly refers to PortA instead of PortB. At present, we wipe all entries from the ports_vulnerable table each time we process a new vuln.xml file. Thus, we start from nil each time. If we are now processing just one vuln at a time, we may have a lingering entry in ports_vulnerable which should be deleted.

How do we detect this situation?

Well, when we process the new vuln, we will delete the old one first, from the vuxml table. When this happens, various triggers are set off to delete related records. We can harness that to delete entries in ports_vulnerable as well.

And how?

Via this table:

freshports.org=# \d commit_log_ports_vuxml
                             Table "public.commit_log_ports_vuxml"
    Column     |  Type   |                              Modifiers
---------------+---------+---------------------------------------------------------------------
 id            | integer | not null default nextval('commit_log_ports_vuxml_id_seq'::regclass)
 commit_log_id | integer | not null
 port_id       | integer | not null
 vuxml_id      | integer | not null
Indexes:
    "commit_log_ports_vuxml_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
    "$1" FOREIGN KEY (vuxml_id) REFERENCES vuxml(id) ON UPDATE CASCADE ON DELETE CASCADE
    "$2" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE CASCADE ON DELETE CASCADE
    "$3" FOREIGN KEY (commit_log_id) REFERENCES commit_log(id) ON UPDATE CASCADE ON DELETE CASCADE

freshports.org=#

When the vuxml entry is deleted, we can see all the ports affected by this vuln. We can then delete entries from the ports_vulnerable table as well.

Or so I think….

Posted in Development | No Comments »

Upgrades over the weekend..

January 23rd, 2012

A number of new features have recently gone into FreshPorts. Each of which has been written above before. I will outline the changes, and provide a link to the original post about that feature.

  • svn coming soon… - FreshPorts now handle commit from three mailing lists: cvs-doc, cvs-ports, and svn-src-all.
  • What ports are dependant upon this port? - this feature has been available on beta and dev for some time. However, reading the comments for that post, it appears we are missing some items from the possible depends that we can pull in.

NOTE: the system is right now updating all the port dependencies. This might take a while… Your port may be out of date right now..

Posted in Development | No Comments »

The committer field is not always an email address

November 1st, 2011

This issue was brought to my attention by wxs (Wesley Shields). The issue is evident at http://www.freshports.org/graphs2.php. When you get there, click on click on Commits Over Time by Committer and then wait for the page to load.

Scroll down to the D’s. There you’ll see a few committers suffixed with ‘(ports committer)’.

Looking in the database, we see:

$ psql freshports.org
psql (8.4.7)
Type "help" for help.

freshports.org=# select committer, message_id, commit_date from commit_log where committer like '%ports committer%' order by commit_date desc;
        committer         |                 message_id                  |      commit_date
--------------------------+---------------------------------------------+------------------------
 rea (ports committer)    | 201110280603.p9S63cVj045250@svn.freebsd.org | 2011-10-28 07:03:38+01
 eadler (ports committer) | 201110172131.p9HLV3xL041492@svn.freebsd.org | 2011-10-17 22:31:03+01
 eadler (ports committer) | 201110161430.p9GEUTFj066335@svn.freebsd.org | 2011-10-16 15:30:29+01
 eadler (ports committer) | 201110152106.p9FL684c030644@svn.freebsd.org | 2011-10-15 22:06:08+01
 rakuco (ports committer) | 201110132036.p9DKaiNi031421@svn.freebsd.org | 2011-10-13 21:36:44+01
 crees (ports committer)  | 201110081825.p98IP22D073560@svn.freebsd.org | 2011-10-08 19:25:02+01
 wxs (ports committer)    | 201110061613.p96GDlZk068019@svn.freebsd.org | 2011-10-06 17:13:47+01
 eadler (ports committer) | 201109282142.p8SLgDCV090941@svn.freebsd.org | 2011-09-28 22:42:13+01
 eadler (ports committer) | 201109282046.p8SKkqQT089293@svn.freebsd.org | 2011-09-28 21:46:52+01
 eadler (ports committer) | 201109281856.p8SIu2n2085649@svn.freebsd.org | 2011-09-28 19:56:02+01
 eadler (ports committer) | 201109281849.p8SInbrJ085366@svn.freebsd.org | 2011-09-28 19:49:37+01
 crees (ports committer)  | 201109281703.p8SH3n8r082164@svn.freebsd.org | 2011-09-28 18:03:49+01
 cs (ports committer)     | 201109162257.p8GMvowd042057@svn.freebsd.org | 2011-09-16 23:57:50+01
(13 rows)

freshports.org=#

Looking at one of these commits (the one by wxs), we see it is a source commit, as opposed to a port commit.

The original commit email contains that ‘(ports committer)’ text.

Ideally, the FreshPorts software would strip this value from the committer field before storing it in the database. It could also set a flag on that commit to highlight this particular situation.

It seems the fix is easy enough:

freshports.org=# update commit_log set committer = replace(committer, ' (ports committer)', '')  where committer like '%ports committer%';
UPDATE 13

But it seems the problem is more widespread than that:

freshports.org=# select committer, message_id, commit_date from commit_log where committer like '% committer%' order by commit_date desc;
       committer        |                 message_id                  |      commit_date
------------------------+---------------------------------------------+------------------------
 wblock (doc committer) | 201110140034.p9E0Yks5038994@svn.freebsd.org | 2011-10-14 01:34:46+01
 gjb (doc committer)    | 201110101123.p9ABNnDV057799@svn.freebsd.org | 2011-10-10 12:23:49+01
 gjb (doc committer)    | 201110101123.p9ABNJuE057749@svn.freebsd.org | 2011-10-10 12:23:19+01
 gjb (doc committer)    | 201110101115.p9ABFvON057412@svn.freebsd.org | 2011-10-10 12:15:57+01
 gjb (doc committer)    | 201110101114.p9ABEiVK057324@svn.freebsd.org | 2011-10-10 12:14:44+01
 gjb (doc committer)    | 201110101114.p9ABE3go057270@svn.freebsd.org | 2011-10-10 12:14:03+01
 gjb (doc committer)    | 201110101112.p9ABCejq057184@svn.freebsd.org | 2011-10-10 12:12:40+01
 gjb (doc committer)    | 201110101111.p9ABBx3e057123@svn.freebsd.org | 2011-10-10 12:11:59+01
 gjb (doc committer)    | 201110101111.p9ABB5M8057060@svn.freebsd.org | 2011-10-10 12:11:05+01
 gjb (doc committer)    | 201110101107.p9AB7anG056904@svn.freebsd.org | 2011-10-10 12:07:36+01
 gjb (doc committer)    | 201110101106.p9AB6nRm056835@svn.freebsd.org | 2011-10-10 12:06:49+01
 gjb (doc committer)    | 201110101105.p9AB5wIj056772@svn.freebsd.org | 2011-10-10 12:05:58+01
 gjb (doc committer)    | 201110092021.p99KL7kn025209@svn.freebsd.org | 2011-10-09 21:21:07+01
 gjb (doc committer)    | 201110021605.p92G5JXb070257@svn.freebsd.org | 2011-10-02 17:05:19+01
 gjb (doc committer)    | 201110012347.p91NlbkK038750@svn.freebsd.org | 2011-10-02 00:47:37+01
 gjb (doc committer)    | 201110012056.p91KuwwX033481@svn.freebsd.org | 2011-10-01 21:56:58+01
 wblock (doc committer) | 201109290337.p8T3bgwv002749@svn.freebsd.org | 2011-09-29 04:37:42+01
 gjb (doc committer)    | 201109290257.p8T2v86W001175@svn.freebsd.org | 2011-09-29 03:57:08+01
 gjb (doc committer)    | 201109290252.p8T2qXbO000985@svn.freebsd.org | 2011-09-29 03:52:33+01
 gjb (doc committer)    | 201109290229.p8T2TWYt000206@svn.freebsd.org | 2011-09-29 03:29:32+01
 gjb (doc committer)    | 201108310118.p7V1IN05074751@svn.freebsd.org | 2011-08-31 02:18:23+01
 gjb (doc committer)    | 201108310117.p7V1Hnam074693@svn.freebsd.org | 2011-08-31 02:17:49+01
 marck (doc committer)  | 201108301149.p7UBnMHr050082@svn.freebsd.org | 2011-08-30 12:49:22+01
 marck (doc committer)  | 201108301147.p7UBla7E049983@svn.freebsd.org | 2011-08-30 12:47:36+01
 gjb (doc committer)    | 201108241218.p7OCITW1053854@svn.freebsd.org | 2011-08-24 13:18:29+01
 marck (doc committer)  | 201108232025.p7NKPBBR018175@svn.freebsd.org | 2011-08-23 21:25:11+01
(26 rows)

freshports.org=#

Posted in Bug fixes, Development, PostgreSQL | No Comments »

Fixes for vuxml processing

October 2nd, 2011

In my previous post, I mentioned problems encountered with processing vuxml entries for horde4-imp. The problem was not specific to a particular port, but came to light because a vuln was registered for a port, and then later removed. I tracked down the cause to not clearing out statistics before processing the new vulnerabilities. This post shows the fix I have come up with.

The table which was not being updated properly was the ports_vulnerable table. This table looks like this:

freshports.org=# \d ports_vulnerable
    Table "public.ports_vulnerable"
 Column  |  Type   |     Modifiers
---------+---------+--------------------
 port_id | integer | not null
 current | integer | not null default 1
 past    | integer | not null default 0
Indexes:
    "ports_vulnerable_pkey" PRIMARY KEY, btree (port_id)
Foreign-key constraints:
    "$1" FOREIGN KEY (port_id) REFERENCES ports(id) ON UPDATE RESTRICT ON DELETE CASCADE

freshports.org=#

The issue was easily resolved with this patch:

--- scripts/process_vuxml.pl	2006/12/17 12:04:02	1.2
+++ scripts/process_vuxml.pl	2011/10/02 17:29:20	1.3
@@ -1,6 +1,6 @@
 #!/usr/bin/perl -w
 #
-# $Id: process_vuxml.pl,v 1.2 2006/12/17 12:04:02 dan Exp $
+# $Id: process_vuxml.pl,v 1.3 2011/10/02 17:29:20 dan Exp $
 #
 # Copyright (c) 2001-2004 DVL Software
 #
@@ -36,7 +36,13 @@ sub EmptyVuXML($) {
 	$sql = "DELETE FROM vuxml";
 	$sth = $dbh->prepare($sql);
 	if (!$sth->execute())  {
-		FreshPorts::Utilities::ReportError('warning', "Could not execute sql", 1);
+		FreshPorts::Utilities::ReportError('warning', "Could not execute sql: " . $sql, 1);
+	}
+
+	$sql = "DELETE FROM ports_vulnerable";
+	$sth = $dbh->prepare($sql);
+	if (!$sth->execute())  {
+		FreshPorts::Utilities::ReportError('warning', "Could not execute sql: " . $sql, 1);
 	}
 }

With patch, I’m clearing out the table each time a new vuln.xml file is processed. However, that is not enough. Whenever a record is removed from this table, the cache for that port needs to be updated.

The following post will queue up a cache deletion request for each delete on the ports_vulnerable table:

--- database-schema/ri.txt	2008/09/09 13:02:38	1.45
+++ database-schema/ri.txt	2011/10/02 17:30:40	1.46
@@ -1,5 +1,5 @@
 --
--- $Id: ri.txt,v 1.45 2008/09/09 13:02:38 dan Exp $
+-- $Id: ri.txt,v 1.46 2011/10/02 17:30:40 dan Exp $
 --
 -- Copyright (c) 1998-2007 DVL Software Limited
 --
@@ -254,6 +254,42 @@ CREATE TRIGGER ports_clear_cache
     EXECUTE PROCEDURE ports_clear_cache();

+CREATE OR REPLACE FUNCTION ports_vulnerable_delete_clear_cache() RETURNS TRIGGER AS $$
+   DECLARE
+      l_cache_clearing_ports_id   int8;
+      l_port      text;
+      l_category  text;
+   BEGIN
+		IF TG_OP = 'DELETE' THEN
+         SELECT port_id
+           INTO l_cache_clearing_ports_id
+           FROM cache_clearing_ports
+          WHERE port_id = OLD.port_id;
+
+          IF NOT FOUND THEN
+            SELECT category, name
+              INTO l_category, l_port
+              FROM ports_all
+             WHERE id = OLD.port_id;
+            INSERT INTO cache_clearing_ports (port_id, category, port)
+            VALUES (OLD.port_id, l_category, l_port);
+          END IF;
+
+          NOTIFY port_updated;
+    END IF;
+
+      -- when a port changes, add an entry to the cache clearing table
+      RETURN OLD;
+   END
+$$ LANGUAGE 'plpgsql';
+
+  DROP TRIGGER ports_vulnerable_delete_clear_cache ON ports_vulnerable;
+CREATE TRIGGER ports_vulnerable_delete_clear_cache
+    AFTER DELETE on ports_vulnerable
+    FOR EACH ROW
+    EXECUTE PROCEDURE ports_vulnerable_delete_clear_cache();
+
+

I’ve tried this in dev, and in production, and staging. It seems to be the right thing to do.

Posted in Development | No Comments »

mail/horde4-imp vulnerability

September 25th, 2011

Mr Thomas wrote me regarding mail/horde4-imp and a particular vulnerability. Specifically, he figured that a recent commit (link to mail archives) fixed a bad entry in vuxml. Let’s look into that now.

I first compared dev, beta, and production. All three had the port flagged as vulnerable. This situation is indicated by a skull at the top of the page, in the ‘Port details’ section. However, only production listed any particular revisions/commits as being vulnerable. This is indicated by skull beside the commit date / revision number. Production also listed versions > 5.0.3 as vulnerable. None of this matches the definition of the vulnerability at the vuxml site, which clearly states:

  • 4.2,1 < horde-imp < 4.3.8,1
  • horde-imp < 4.3.8

Let’s look at vuln.xml:

<name>horde-imp</name>
<range><gt>4.2,1</gt><lt>4.3.8,1</lt></range>
<range><lt>4.3.8</lt></range>

The next check: has FreshPorts processed the latest vuxml file?

Clicking on the skull takes us to http://www.freshports.org/vuxml.php?package=horde-imp, where you can see:

The last vuln.xml file processed by FreshPorts is:

    Revision:  1.2451
    Date:      2011/09/23
    Time:      20:02:19
    Committer: delphij

If we compare that to the most recent commit for security/vuxml, we can confirm the revision number. Yes, that matches up.

That looks right.

OK, what ports have a package name of horde-imp?

freshports.org=# select id, category || '/' || name as port from ports_active where package_name = 'horde-imp';
  id   |      port
-------+-----------------
 21107 | mail/horde-imp
 28743 | mail/horde4-imp
(2 rows)

freshports.org=#

This confirms that this vuxml entry affects both horde4-imp and horde-imp. However, we are only interested in id = 28743.

Looking in the log for the vuxml processing, I found nothing to indicate that 28743 is vulnerable to anything. Up to this point, I was doing everything on the dev server. Next, I went to the production server and checked the vuxml log. Nothing there either. OK, so perhaps this is something else entirely.

Next, I went to production and cleared the cache for all mail/horde* ports. Ahah! Now prod, dev, and test are in agreement.

The only issue now is the vulnerability skull at the top of the page. This usually indicates that a vulnerability affects the latest revision of the port. Hmmm, this needs further investigation.

This query shows that FreshPorts thinks there is a current vulnerability:

freshports.org=# select * from ports_vulnerable where port_id = 28743;
 port_id | current | past
---------+---------+------
   28743 |       1 |    0
(1 row)

freshports.org=#

As a test, I reran, with a commit, the adjusting function:

freshports.org=# select * from ports_vulnerable where port_id = 28743;
 port_id | current | past
---------+---------+------
   28743 |       1 |    0
(1 row)

freshports.org=# begin;
BEGIN
freshports.org=# select PortsVulnerabilityCountAdjust(28743);
 portsvulnerabilitycountadjust
-------------------------------
                             0
(1 row)

freshports.org=# select * from ports_vulnerable where port_id = 28743;
 port_id | current | past
---------+---------+------
(0 rows)

freshports.org=# rollback;
ROLLBACK
freshports.org=# select * from ports_vulnerable where port_id = 28743;
 port_id | current | past
---------+---------+------
   28743 |       1 |    0
(1 row)

freshports.org=#

My conclusion: PortsVulnerabilityCountAdjust() was not invoked for this port.

The vuxml log confirms this.

Offhand, I think the solution is to delete everything from ports_vulnerable each time we process a new vuxml file.

That’s all for tonight. I want to think about this for a while longer.

Posted in Bug fixes, Development, PostgreSQL, vuxml | No Comments »

commits without files

September 24th, 2011

In short, the concept of any computer system is usually straight forward. The complexity comes from the exceptions. Take commits for example. Yes, each commit affects a file. Or so you’d think. Consider this commit.

FreshPorts parses the email. In that email, there are no files mentioned. This caused a problem during the loading. FreshPorts, as a sanity test, complained there were no files. This complaint is now just a notification, not a fatal error.

The fix for that was rather simple:

-		FreshPorts::Utilities::ReportError('Err', "No files found in commit '$Updates{MessageId}'.  Has someone done a cvs import instead of addport?", 1)
+		FreshPorts::Utilities::ReportError('Err', "No files found in commit '$Updates{MessageId}'.  Has someone done a cvs import instead of addport?", 0)

In this case, the 1 means die, fatal error. Hmmm, wouldn’t that be better with a constant?

The next issue was the display of the commit. Production was displaying the commit. Dev and beta was not. Eventually I figured out the problem was a long-standing stored procedure.

The fix for that issue was much more subtle:

@@ -2185,13 +2185,14 @@ SELECT 0 AS port_id,
        NULL::text AS only_for_archs,
        NULL::text AS not_for_archs,
        OnWatchList($4, CLE.element_id) AS watch
-  FROM commit_log CL, commit_log_elements CLE, element E
- WHERE CL.id          = CLE.commit_log_id
-   AND CL.message_id  = $1
-   AND CLE.element_id = E.id
+  FROM commit_log CL LEFT OUTER JOIN commit_log_elements CLE ON (CL.id = CLE.commit_log_id)
+       LEFT OUTER JOIN element E ON (CLE.element_id = E.id)
+ WHERE CL.message_id  = $1
 ORDER BY port, element_pathname
    LIMIT $2
   OFFSET $3;
+
+  -- NOTE some commits touch nothing....  e.g. 201109230051.p8N0pbV2045995@svn.freebsd.org

 $$ LANGUAGE SQL STABLE;

This is the code for retrieving a commit. It also retrieves the files touched by that commit. Thus, if there are no files, it will not retrieve anything. Thereby, incorrectly reporting ‘Sorry, nothing found in the database….’.

The old code assumed that for every entry (commit) in the commit_log table, there would be at least one element (file). The SQL change made entries in the commit_log_elements and the element tables would be optional.

This commit now displays correctly in all three environments:

  1. Production
  2. Beta
  3. dev

But why was production displaying the commit? Because it still contains older code which has since been fixed. This fix stopped FreshPorts from incorrectly treating ‘Directory Properties:’ and ‘(props changed)’ as ‘files”. The fix was on beta and dev. Hence, those systems had no entries in commit_log_elements for those bogus files. This is why they would not display the commit.

But now I have bogus files sitting around. I deleted them by hand.

Then I noticed that commit_log_elements was missing foreign keys. I added them, but first had to delete rows which referred to invalid references. These two commands took care of that:

delete from commit_log_ports_elements where not exists (select * from commit_log where commit_log_id = id);
delete from commit_log_ports_elements where not exists (select * from element where element_id = id);

Then I added two new foreign keys:

alter table commit_log_ports_elements
    add foreign key (element_id)
       references element(id) on update cascade on delete cascade;

alter table commit_log_ports_elements
    add foreign key (commit_log_id)
       references commit_log(id) on update cascade on delete cascade;

The table in question now looks like this:

freshports.org=# \d commit_log_elements
                               Table "public.commit_log_elements"
    Column     |     Type     |                            Modifiers
---------------+--------------+------------------------------------------------------------------
 id            | integer      | not null default nextval('commit_log_elements_id_seq'::regclass)
 commit_log_id | integer      | not null
 element_id    | integer      | not null
 revision_name | text         |
 change_type   | character(1) | not null
Indexes:
    "commit_log_elements_pkey" PRIMARY KEY, btree (id)
    "commit_log_elements_clid" btree (commit_log_id)
    "commit_log_elements_ei" btree (element_id)
Check constraints:
    "commit_log_elements_change_type" CHECK (change_type = 'A'::bpchar OR change_type = 'M'::bpchar OR change_type = 'R'::bpchar)
Foreign-key constraints:
    "$1" FOREIGN KEY (commit_log_id) REFERENCES commit_log(id) ON UPDATE CASCADE ON DELETE CASCADE
    "$2" FOREIGN KEY (element_id, revision_name) REFERENCES element_revision(element_id, revision_name) ON UPDATE CASCADE ON DELETE CASCADE
Referenced by:
    TABLE "commit_log_port_elements" CONSTRAINT "$2" FOREIGN KEY (commit_log_element_id) REFERENCES commit_log_elements(id) ON UPDATE CASCADE ON DELETE CASCADE
Triggers:
    element_delete_check BEFORE INSERT OR UPDATE ON commit_log_elements FOR EACH ROW EXECUTE PROCEDURE element_delete_check()

Now I should be able to get into other things. :)

Posted in Development | No Comments »

svn coming soon…

September 17th, 2011

A while back, the src tree for FreeBSD moved to svn. The ports tree is still on cvs. For some time now, http://svn.freshports.org/ has been including the svn commits. But now, as old mailing lists become inactive, it’s time to start integrating things into the main websites.

The dev and beta websites have been running on three different mailing lists and two repository types for a few months now. This sounds complicated, but because all of FreshPorts works off an XML-based input, all that’s needed for input is a script to convert the email commit to the XML format. From there, all commits are the same to FreshPorts.

FreshPorts now uses three mailing lists. If you think we should cover more, we can.

  1. cvs-doc : CVS commit messages for the doc and www trees
  2. cvs-ports : CVS commit messages for the ports tree
  3. svn-src-all : SVN commit messages for the entire src tree (except for “user” and “projects”)

Tonight I’ll be setting up a new FreshPorts server and going through the upgrade process to make sure all goes well.

Posted in Development | No Comments »

« Previous Entries