We’re still recovering from our epic outage, which lasted from about 8pm PST September 4th until 4am PST September 6th. By several magnitudes, this was the longest downtime we’ve had in our 7 years of publishing. That shit was brutal. After troubleshooting on the old server for 20 hours, we decided to abandon ship and move to a brand new server. So far, it’s sparkly and doing well.
I’ve been using the Offbeat Empire’s Facebook page to keep everyone updated, but now that offbeatempire.com is back online (on an entirely new server, built from scratch) I’ll update this post with updates as we have them.
Current status, as of 6am PST September 6…
- Sleep status: our server tech Chris was literally up on all night. Kellbot’s on 3hrs of sleep.
- Server status: we are on a completely new server. (See photo illustration of what I imagine it looks like.)
- Website status: Offbeat Bride is up, but lots of functions are inactive. Offbeat Home and Offbeat Families and the Tribe are intentionally being kept down until we make sure things are fully stable on the new server.
We’re getting there! Big thanks to Chris and Kellbot for burning the midnight oil…
We’re slowly re-enabling functions on Offbeat Bride this morning, waiting 5 minutes between each function to make sure the server stays stable. Once we have Offbeat Bride (the flagship and cash cow of the Empire) fully functional and stable, we can start enabling the other sites.
Our goal is to have Home & Families up by 9am PST.
The Offbeat Bride Tribe is the most complex site we have, so it’ll be longer… probably not until this evening.
Thanks everyone for their patience and to my developer Kellbot and the crew at our host Liquid Web for pulling all-nighters… this has been rough.
Update 8am PST
Honestly, the best thing y’all can do right now is stay off the sites. 🙁 Our server is EXTREMELY delicate, and we need to come back online slowly. So let’s go look at adorable pictures on Flickr, hmm?!
Update 10am PST
Ooh, things are looking good! You can stop holding your clicks (or dicks, or clits, or whatever else you were holding) and just browse Offbeat Bride like normal. We’re watching things closely, and getting ready to start working on Home and Families soon.
Update 11am PST
At a reader’s suggestion, we have started a donation fund so y’all can pitch in for some premium whiskey to thank our Liquid Web server techs who worked straight through the night.
Whiskey fund is closed: we’ll be delivering it on Tuesday!
Update 12:30pm PST
Offbeat Home and Offbeat Families are online! Our new server is holding firm, and things are looking great. This means that we’ve now got all sites up — except for the Offbeat Bride Tribe. The sad news is that given how complex the Tribe is, and the fact that I want my developer Kellbot to actually have a weekend after working straight for 20 hours… we’re not going to be able to get the Tribe up until Monday. 🙁 We’ve posted a Tribesmaid support group open thread so Tribesmaids can figure out how they’re going to survive the weekend.
And thanks to all of you who’ve chipped in on the whiskey donation! Ingvild, Emily, Article Seven, Jen, Amanda, Alissa, Holly, and Kirsty? YOU GUYS ARE FUCKING AWESOME.
Update 2:30p PST
So, did anyone notice anything different with the homepages of the Empire sites? We hit a bug that we decided to workaround for now by changing the design… anyone spotted it yet?
Update Monday September 9, 7am PST
The Tribe went back up a couple hours ago, and with that, we’re officially COMPLETELY back up from our outage! Whew. Our new server is doing great, and we’re all freaking relieved.
The homepage design tweak that no one was able to spot is this: It used to be on our homepages, we did one big post at the top, and then small left-aligned thumbnails and post excerpts for the rest of the posts on the page page. Now, we’re doing big posts for the ENTIRE homepage, and thumbnails + excerpts starting on the Page 2 of the blogs. If you want to compare, click here and then click here.
Also, I’m closing the whiskey fund because we’ve got enough to put together an amazing gift basket which a loving Booze Santa will be delivering to the web server technicians on Tuesday. I’ll have pictures to share as a thank you to all our amazing whiskey fund donors!
