FreshPorts downtime

Earlier this week, we had a bit of a crisis. The colo which provides bandwidth and servers to FreshPorts (and all the other websites I run), was unexpectedly shutdown. BChosting sold their Vancouver facilities to Netluxe, who, by all accounts, shut down the ops center without notifying customers. Well, I wasn’t notified. I was contacted by one other customer and they knew nothing. He said the Netluxe help desk told him things were being moved, but did not provide any other information.

The BChosting folks noticed what had happened. They went managed to rescue some customer equipment (including my box, which I do not own). They have since got that box up and running on the old IP addreses at another colo.

This timing was ironic. I am merely weeks away from deploying the dual opteron. It is sitting here, the shipping paper work is all ready to go. I just have to do some final things, like remove the rails, and call Fedex. Three weeks later, and this situation would not have mattered.

In the meantime, I contacted NYI and asked for for a short term emergency box to tide me over. They did exactly that. FreshPorts is now running from within NYI’s facilities. I had a box within a couple of hours. It took about 4 hours to compile all the required ports. And by the next morning things were up and running. When I asked for more RAM, I wound up with a total of 2GB. This is what I’m using now:

CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2394.01-MHz 686-class CPU)
  Origin = “GenuineIntel” Id = 0xf41 Stepping = 1
  Features=0xbfebfbff
  Features2=0x441d>
real memory = 2129850368 (2031 MB)
avail memory = 2083422208 (1986 MB)

FreshPorts is flying!

It took a few hours to catch up with the queued emails. To speed up the process, I transferred some of the emails, in XML format, from the BETA system and injected them into the queue. No commits were lost.

FreshPorts et al will continue to do so until I deply the new server. At which time both the NYI and BChosting will be moved to hot standby. Should anything occur with the main server, all it takes is a DNS change and the new server will be up and running. Slony will be involved to keep the two standby databases up to date.

My thanks to NYI for coming through on short notice. And to BChosting for rescuing the equipment and getting the box back online.

On a positive note, the Makefile I have for installing FreshPorts dependencies came in very handy. I will also be adding a few things to my daily backups:

• /boot
• /path/to/websites/*/configuration

• I was looking for wildcarding features in my backup software, and found bwild, which helps you test out wild card directives against a list of files. That is much more helpful than resorting to running a job to see if you get everything you need. Nice addition to the toolset!

  :)