Virtually all modern 3rd-party / open source PHP applications will run fine under PHP5 without modification.

Update: This issue was fixed by around 6:00 PDT. The spam flow was due to a small - and elusive - misconfiguration in SpamAssassin which neither we nor our datacenter were able to figure out; cPanel experts helped find and correct the problem. We’ve stabilized now on our new operating system and should be in for smooth sailing. Thanks for your patience.

Reminder: If you wish to perform client-side spam filtering to enhance the server-side filtering provided by Birdhouse, see our Spam Control FAQ.

Update Jan. 16, #1: Spam filters are borked once again. We’re doing what we can to fix them.

Update Jan. 16, #2: Spam filtering is now working properly, hopefully for good! Thanks for hanging in there - we hate spam as much as you do.

• Red Hat 9 → CentOS 4
• PHP 4 → PHP 5
• MySQL 4 → MySQL 5

We anticipate up to an hour of downtime, though it should be less than that total. The upgrade will be handled by an automated script, painstakingly written and watched over by our rockin’ datacenter. The process is well-tested and we do not anticipate any major issues, though it’s possible some minor things may crop up. Please let us know immediately if anything’s not working for you after the upgrade.

Update: The operating system upgrade went extremely well; downtime was less than 30 minutes. We had a few loose ends to tie up: Missing Smarty libs, some SSL issues, but everything’s getting ironed out quickly. SSL email on gong.birdhouse.org may not be fixed before morning.

SpamAssassin kept running, but the setting that enables SA for each user automatically had not made it across the migration. Spam was allowed through the system during this period. Spam settings were re-enabled a couple of hours after the migration was completed.

Unfortunately, the PHP / Apache / MySQL upgrades may have to wait for another day. We’re still running the previous AMP stack, and will tackle this portion of the upgrade soon.

Update 2:  SSL certs for POP/IMAP access are working again, and we’ve ironed out all known issues  (all were fairly minor;  most users didn’t notice that anything had changed).

We’ll post an update on the AMP upgrade plans soon.

Our Configure Spam Controls FAQ has been updated accordingly. If you were maintaining any server-side spam filters or rules, please see the FAQ, then log into cPanel and take a look to make sure your mail is being handled the way you want it to be.

The likelihood of these two events happening simultaneously is like lightning hitting twice in the same spot. Guess we’re just lucky :).

MailScanner has been removed from Birdhouse and replaced with the original (and more straightforward) cPanel + SpamAssassin system. This change will result in both lighter loads and easier troubleshooting of our mail system in the event of any future problems.

When MailScanner was first installed, it offered a lot of functionality not offered by cPanel + SpamAssassin alone. But with the move to cPanel 11, enough new features have been added that we no longer need MailScanner. Removing MailScanner also clears the path for us to move from the current operating system to CentOS 4.x in the near future.

Update Jan. 2, 2008: Making progress — we’ve scripted a solution to remove spam and misdirected messages from the queue. The script is running now and making a good dent in the load. Unfortunately, it looks like it will take a while for the script to complete. Mail should start trickling in soon. We’ll keep you updated.

Update Jan. 3, 2008: We’ve found a way to tremendously speed up the queue processing, and are making our way through the backlog very quickly now. We expect all mail to be delivered by the morning of Jan. 3. However, we’ve had to let a fair bit of spam through the filters to make it happen. At this point, we felt it was better to make sure people got their mail even if it included a lot of spam than to delay delivery unnecessarily.

Update #2 - Jan. 3, 2008: We’ve completed the main processing on the queue. However, there are still 50,000 messages in the primary queue, and MailScanner is currently broken. New mail is arriving normally (and so is spam) but the backlogged mail is currently trapped. A consultant is working on this problem with us.

Update Jan. 4, 2008: The backlogged queue has been unfrozen and is now being processed normally. It will take a while for everything to move through the system, but the delivery rate is good. MailScanner has been disabled and cPanel spam controls have changed. We’ll update the FAQs to reflect the changes soon. Note: The backlog will NOT be spam filtered, so users should expect to continue receiving more than normal spam levels until it’s finished. NEW incoming mail is being spam filtered, however.

Update #2, Jan. 4, 2008: Looks like the backlog will be completely processed by sometime late Friday night.

Update #3, Jan. 4, 2008: As of now, we have a single mail  queue, which is currently down to 25,000 messages (it was at 50,000 last night). The queue is not frozen, and users are seeing mail from  the past week trickle slowly into their mailboxes. We expect the queue size to return to normal - and for all outstanding mail to have been delivered - by Saturday morning.

Newly arriving mail is being delivered immediately and is being spam-processed. Older mail already in the queue is trickling out and is not being spam processed.  Once the queue has been emptied, all spam and delivery levels will return to normal.

 Update: Two minutes later… problem solved