Birdhouse Hosting System Status » General

Datacenter network issue

shacker — Sat, 05 Sep 2009 18:59:58 +0000

Birdhouse servers are completely offline right now – our datacenter is experiencing a network hardware issue affecting dozens of servers and thousands of clients (it’s not just us). Please stay tuned for updates.

Update: We came back online around 2:00 pm Pacific time. Looks like this was a cooling issue at the datacenter. Thanks for your patience everyone.

MySQL Upgrade

shacker — Sun, 26 Jul 2009 06:38:34 +0000

The MySQL database engine on Birdhouse has been upgraded from version 4 to version 5. Sites were offline for just a few minutes during the transition. Please let us know if you encounter any issues.

Apache Downtime

shacker — Tue, 07 Apr 2009 19:25:34 +0000

All web services are temporarily down, as we compile some additional modules and options into Apache. We apologize for the unscheduled downtime. We’re working to bring service back online as soon as possible.

Update 12:30 pm PDT: All systems are back up and working normally. Again, apologies for the inconvenience.

SSL Certificate Confusion

shacker — Sat, 21 Jun 2008 21:18:35 +0000

While our SSL certificate is currently valid and current, the server suddenly started handing out a self-signed certificate instead. This means users will likely be seeing “untrusted certificate” warnings from their web browsers (when connecting to webmail or cpanel) or from their email clients when using POP or IMAP connections. We’re currently working with the datacenter on getting this straightened out. We apologize for the inconvenience.

Note that it IS safe to accept the self-signed certificate and continue with your work.

Apache and PHP Upgrades Complete

shacker — Mon, 21 Apr 2008 00:21:34 +0000

We have completed the process of upgrading Apache and PHP to the latest versions, and everything appears to be running normally. In rare instances, some home-brew PHP scripts may break in unexpected ways. These breakages are generally quite easy to fix – contact us and we’ll take a look.

Virtually all modern 3rd-party / open source PHP applications will run fine under PHP5 without modification.

emacs installed

shacker — Wed, 13 Feb 2008 18:15:35 +0000

On customer request, emacs is now installed and working (vi and pico were already present). Go for it!

Loads Are Light

shacker — Wed, 06 Feb 2008 08:22:25 +0000

Quick update: Since migrating from RedHat 9 to CentOS 4.6 and removing MailScanner a few weeks ago, load averages on Birdhouse servers have been reduced almost by a factor three. Performance has been excellent, and all systems are humming along nicely.

Spam (Again)

shacker — Sat, 12 Jan 2008 00:00:28 +0000

SpamAssassin fell over this morning and spam is finding its way into customer mailboxes. We’re working with the datacenter on a fix. Please stand by.

Update: This issue was fixed by around 6:00 PDT. The spam flow was due to a small – and elusive – misconfiguration in SpamAssassin which neither we nor our datacenter were able to figure out; cPanel experts helped find and correct the problem. We’ve stabilized now on our new operating system and should be in for smooth sailing. Thanks for your patience.

Reminder: If you wish to perform client-side spam filtering to enhance the server-side filtering provided by Birdhouse, see our Spam Control FAQ.

Update Jan. 16, #1: Spam filters are borked once again. We’re doing what we can to fix them.

Update Jan. 16, #2: Spam filtering is now working properly, hopefully for good! Thanks for hanging in there – we hate spam as much as you do.

Major System Update Scheduled

shacker — Tue, 08 Jan 2008 02:18:12 +0000

On Tuesday, Jan. 8, at 7pm Pacific time, we’ll be undertaking a major technical upgrade of our hosting platform:

Red Hat 9 → CentOS 4
PHP 4 → PHP 5
MySQL 4 → MySQL 5

We anticipate up to an hour of downtime, though it should be less than that total. The upgrade will be handled by an automated script, painstakingly written and watched over by our rockin’ datacenter. The process is well-tested and we do not anticipate any major issues, though it’s possible some minor things may crop up. Please let us know immediately if anything’s not working for you after the upgrade.

Update: The operating system upgrade went extremely well; downtime was less than 30 minutes. We had a few loose ends to tie up: Missing Smarty libs, some SSL issues, but everything’s getting ironed out quickly. SSL email on gong.birdhouse.org may not be fixed before morning.

SpamAssassin kept running, but the setting that enables SA for each user automatically had not made it across the migration. Spam was allowed through the system during this period. Spam settings were re-enabled a couple of hours after the migration was completed.

Unfortunately, the PHP / Apache / MySQL upgrades may have to wait for another day. We’re still running the previous AMP stack, and will tackle this portion of the upgrade soon.

Update 2: SSL certs for POP/IMAP access are working again, and we’ve ironed out all known issues (all were fairly minor; most users didn’t notice that anything had changed).

We’ll post an update on the AMP upgrade plans soon.

MailScanner Removed

shacker — Sat, 05 Jan 2008 00:41:05 +0000

In addition to the enormous mail backlog that landed on our doorstep last week, we simultaneously hit an additional snag: An update of perl to v5.8.8 broke MailScanner and could not be fixed without a major operating system upgrade. But the operating system upgrade cannot be undertaken with MailScanner installed – Catch 22 (documentation on this issue is here). As a result, we had an enormous backlog of mail that simply couldn’t be released quickly.

The likelihood of these two events happening simultaneously is like lightning hitting twice in the same spot. Guess we’re just lucky .

MailScanner has been removed from Birdhouse and replaced with the original (and more straightforward) cPanel + SpamAssassin system. This change will result in both lighter loads and easier troubleshooting of our mail system in the event of any future problems.

When MailScanner was first installed, it offered a lot of functionality not offered by cPanel + SpamAssassin alone. But with the move to cPanel 11, enough new features have been added that we no longer need MailScanner. Removing MailScanner also clears the path for us to move from the current operating system to CentOS 4.x in the near future.