Earlier today the FreshBooks service experienced approximately half an hour of downtime.

At approximately 11:37 AM Eastern the FreshBooks website and application were not available. We are not happy to report this and before I explain what happened, I would like to apologize to anyone that was affected.

Today’s downtime was the result of human error configuring routers at Rackspace while they were moving our old database servers to the new cabinet, compounded by miscommunication between our account manager and their network technicians and problems with Rackspace’s phone system.

Timeline of events (EDT):

Rackspace started move: 11:26
Website unavailable: 11:37
Workaround in place: 12:05
Root cause resolved: 13:00 approx

(plus about a minute at around 12:45 to redo the workaround).

The workaround was an SSH tunnel from our database servers to our web servers on one of the vhost addresses that was still working. (No traffic could reach our database servers, but some traffic could reach some addresses on our webservers from our database servers).

In future, we’ll be scheduling changes in which Rackspace has the ability to cause collateral damage after hours, even when those changes don’t affect production systems.

Again, we are sorry for the inconveniences this caused, and if you were adversely affected by the downtime, we invite you to please send us a note or give us a call so that we can hear from you and look into making it up to you.

Thank you for your understanding.

For future reference, we have a status blog at http://status.freshbooks.com which gives real-time updates of the FreshBooks service.