Deterring bots, but letting people in

This applies to upcoming webserver changes. When you see a date in this paragraph, it has been deployed.

The search gate was meant to keep bots away from the search page while letting people in without a login. A botnet got past it, and the database ran at 100% CPU until search was restricted to logged-in users. This post covers how it got past, and the two changes that close the gap: one in the gate, one in nginx.

How the gate was bypassed

The gate let a search through if the visitor was logged in, had a signed search_ok cookie, or came from a page on this site, judged by the Referer header. That last rule was meant for people clicking the maintainer, committer, and “depends on this port” links on port pages.

The bot sent a Referer naming a FreshPorts category page on every request, and walked through “depends on this port” searches, the most expensive kind. Each request came from a different residential IP address, so blocking addresses would not have helped. robots.txt already disallows the search pages.

The same bot also requested commit.php and vuxml.php, which require a login. Those requests were refused, but only after PHP had started and opened a database connection.

The changes

The gate no longer trusts the Referer

A search now needs a login or a valid search_ok cookie. Anyone else sees a page with a button; pressing it sets the cookie for a day and returns them to their search. People following a link from a port page see that page once a day. Logged-in users never see it.

The challenge page is now also served by www/search-gate.php on a GET. That script loads no database code, which is what lets nginx hand bots to it.

nginx stops these requests before PHP

• commit.php and vuxml.php: without a login cookie, nginx returns 401 itself.
• search.php: a search (a request with a query) without a login or search_ok cookie is passed internally to search-gate.php, which shows the challenge page. The empty search form still loads for anyone.

nginx only checks that a cookie is present; it cannot check the signature. PHP still verifies everything that gets through, so a made-up cookie gets the same answer as before, just after a database connection.

Deployment

Deploy the PHP first: nginx sends requests to search-gate.php with a GET, which the old version did not handle.

1. Deploy the code

• include/search-gate.php: the Referer check is gone, and the challenge page is its own function.
• www/search-gate.php: shows the challenge page on a GET. With the gate off, it returns 401 instead of redirecting, so it cannot loop with nginx.

2. Turn the gate on

In include/constants.local.php, set the secret used to sign the cookie and the button’s token. Generate one with openssl rand -hex 32. Changing it later invalidates everyone’s cookie.

define('SEARCH_GATE_SECRET', '<64 hex characters>');

With the secret set, the gate decides who may search and LOGIN_TO_SEARCH no longer applies to search.

3. nginx

At http level, next to the limit_req_zone lines:

# a search (query present) with no login cookie and no search-gate cookie
map "$cookie_visitor|$cookie_search_ok|$arg_query" $fp_search_needs_gate {
    default      0;
    "~^\|\|.+"   1;
}

In virtualhost-common.conf, before the general location ~ \.php$ block:

  # LOGIN_TO_VIEW_COMMIT / LOGIN_TO_VIEW_VUXML: refuse before PHP and the database
  location ~ ^/(commit|vuxml)\.php$ {
    if ($cookie_visitor = "") { return 401 "You must be logged in to use this feature."; }
    include /usr/local/www/freshports/configuration/php-processing.conf;
  }

  # searches without a login or search_ok cookie get the challenge page
  location = /search.php {
    if ($fp_search_needs_gate) { rewrite ^ /search-gate.php last; }
    include /usr/local/www/freshports/configuration/php-processing.conf;
  }

The rewrite is internal, so there is no extra redirect, and the query string is passed along. If the gate secret is ever removed, search-gate.php returns 401 for these requests, so this configuration is correct whether the gate is on or off.

nginx -t && service nginx reload

Verification

The first five were run against the development site, which has the new gate but not the nginx change. All seven apply to production once both are in place:

Request	Expected
A search with a faked FreshPorts Referer and no cookies	403, the challenge page
GET /search-gate.php?… with no cookies	403, the challenge page, with the search kept for the button
Pressing the button	303 back to the search, with the search_ok cookie set
The same search with that cookie	200, the results
A post with a bad token	303 back to the search, which shows the challenge again
commit.php or vuxml.php with no cookies	401, from nginx
search.php with no query	200, the search form

Then watch the access log for a day. Bot requests to these three pages should show 401 or 403 with small response sizes, and the database should not see them at all.

Rollback

• Close search to visitors who are not logged in: comment out SEARCH_GATE_SECRET. LOGIN_TO_SEARCH takes over again, and the nginx configuration can stay.
• Undo nginx: remove the two location blocks and the map, then nginx -t and reload.
• Undo the code: revert the commit. Do this only after the nginx blocks are gone, since nginx depends on the new search-gate.php.