Posted by
We’re still recovering from our epic outage, which lasted from about 8pm PST September 4th until 4am PST September 6th. By several magnitudes, this was the longest downtime we’ve had in our 7 years of publishing. That shit was brutal. After troubleshooting on the old server for 20 hours, we decided to abandon ship and move to a brand new server. So far, it’s sparkly and doing well.
I’ve been using the Offbeat Empire’s Facebook page to keep everyone updated, but now that offbeatempire.com is back online (on an entirely new server, built from scratch) I’ll update this post with updates as we have them.
Current status, as of 6am PST September 6…
- Sleep status: our server tech Chris was literally up on all night. Kellbot’s on 3hrs of sleep.
- Server status: we are on a completely new server. (See photo illustration of what I imagine it looks like.)
- Website status: Offbeat Bride is up, but lots of functions are inactive. Offbeat Home and Offbeat Families and the Tribe are intentionally being kept down until we make sure things are fully stable on the new server.
We’re getting there! Big thanks to Chris and Kellbot for burning the midnight oil…
We’re slowly re-enabling functions on Offbeat Bride this morning, waiting 5 minutes between each function to make sure the server stays stable. Once we have Offbeat Bride (the flagship and cash cow of the Empire) fully functional and stable, we can start enabling the other sites.
Our goal is to have Home & Families up by 9am PST.
Your website’s auto-play music may be losing you clients
Business owners and wedding vendors, you know I love you, but I have to get honest with you about something: the auto-play music on your... Read more
The Offbeat Bride Tribe is the most complex site we have, so it’ll be longer… probably not until this evening.
Thanks everyone for their patience and to my developer Kellbot and the crew at our host Liquid Web for pulling all-nighters… this has been rough.
Update 8am PST
Honestly, the best thing y’all can do right now is stay off the sites. 🙁 Our server is EXTREMELY delicate, and we need to come back online slowly. So let’s go look at adorable pictures on Flickr, hmm?!
Update 10am PST
Ooh, things are looking good! You can stop holding your clicks (or dicks, or clits, or whatever else you were holding) and just browse Offbeat Bride like normal. We’re watching things closely, and getting ready to start working on Home and Families soon.
Update 11am PST
At a reader’s suggestion, we have started a donation fund so y’all can pitch in for some premium whiskey to thank our Liquid Web server techs who worked straight through the night.
Whiskey fund is closed: we’ll be delivering it on Tuesday!
Update 12:30pm PST
Offbeat Home and Offbeat Families are online! Our new server is holding firm, and things are looking great. This means that we’ve now got all sites up — except for the Offbeat Bride Tribe. The sad news is that given how complex the Tribe is, and the fact that I want my developer Kellbot to actually have a weekend after working straight for 20 hours… we’re not going to be able to get the Tribe up until Monday. 🙁 We’ve posted a Tribesmaid support group open thread so Tribesmaids can figure out how they’re going to survive the weekend.
And thanks to all of you who’ve chipped in on the whiskey donation! Ingvild, Emily, Article Seven, Jen, Amanda, Alissa, Holly, and Kirsty? YOU GUYS ARE FUCKING AWESOME.
Update 2:30p PST
So, did anyone notice anything different with the homepages of the Empire sites? We hit a bug that we decided to workaround for now by changing the design… anyone spotted it yet?
Update Monday September 9, 7am PST
The Tribe went back up a couple hours ago, and with that, we’re officially COMPLETELY back up from our outage! Whew. Our new server is doing great, and we’re all freaking relieved.
The homepage design tweak that no one was able to spot is this: It used to be on our homepages, we did one big post at the top, and then small left-aligned thumbnails and post excerpts for the rest of the posts on the page page. Now, we’re doing big posts for the ENTIRE homepage, and thumbnails + excerpts starting on the Page 2 of the blogs. If you want to compare, click here and then click here.
Also, I’m closing the whiskey fund because we’ve got enough to put together an amazing gift basket which a loving Booze Santa will be delivering to the web server technicians on Tuesday. I’ll have pictures to share as a thank you to all our amazing whiskey fund donors!
May the force be with your server techs.
Glad to see you’re on your way back.
And congrats on the truly fabulous new server.
I keep checking, getting excited when it comes up, and then disappointed when it goes back down again lol. I’m probably part of the problem haha. You guys are amazing for all of the effort getting it back up and we appreciate it! I’m gonna get off now so I don’t make things worst lol but I do appreciate it!!!
Thank you guys for all your hard work.
Whee new server and such. Fingers crossed it holds up.
Can we all chip in to buy your server techs some whiskey or booze of their choice? They deserve it, when it’s all done. I can’t imagine it’s an easy fix.
Seconded.
Laura-Nicole, it is SO funny you should mention this! It’s like you’re eavesdropping on our conversation … just a few hours ago, I asked the server guys what their favorite brand of Whiskey was. And YES. We can totally do a fund… cuz these guys deserve some PREMIUM shit.
Click here to donate. 🙂
The geek in me (which is pretty much all of me) has been holding out hope you’d be able to find out the cause of the mysterious crashing and write up a post-mortem on it. So glad you’re back up, though — and the old server can remain a big old mystery box, like the one in JJ Abrams’s shed.
Oh dude, we LOVE a good post mortem, but it’s part of what has been amazing about this whole thing: we’ve had THREE senior server techs putting in epic shifts (a collective, like 48 hours of labor) trying to figure out what was going on with our old server, host2.
They have all been stumped. Something was deeply fucked with the server, but last I heard from one of the techs (Chris, who literally stayed up all night hammering at this thing) he was opting to change careers to become a farmer… because he was so stumped by wtf was going on with our server.
But yeah: if we can figure it out, we will for sure share. I LOVE a good post mortem.
My advice to Chris: don’t become a farmer if you want to be able to understand what’s going on. Things do well, or don’t do well, and shit’s complicated. Farming is hard.
Not sure if it’s related…probably not, but I’ve noticed for the past 4 or 5 days none of the posts are coming up in my feedly feed. 🙁
It’s absolutely related. We’ve got to get the sites stable first, then we’ll give the RSS feeds a good hard kick. 🙂
Oh good, I’m not crazy or completely inept! That’s nice to know. 🙂
JB, I *think* this has resolved itself — are you now seeing our feeds in your Feedly? (I use Feedly too, and they just showed up for me.)
They never did come through to my Feedly and I have refreshed many times. It looks like the last day was Aug 26 for OBH and OBF. Nothing has come from those 2 since then. I do get the OBE posts, though. Could it be change of IP Address and should I re-RSS the sites?
I don’t think a change of IP would impact it, but I’ll ask the Feedly folks. In the meantime, your best bet is probably unsubscribing and resubscribing to our feeds through Feedly.
Sorry for the hassle!
Just checked in with Feedly about this… go ahead and go here to make sure you’re subscribed:
BRIDE:
http://cloud.feedly.com/#subscription%2Ffeed%2Fhttp%3A%2F%2Foffbeatbride.com%2Ffeed
HOME:
http://cloud.feedly.com/#subscription%2Ffeed%2Fhttp%3A%2F%2Foffbeathome.com%2Ffeed
FAMILIES:
http://cloud.feedly.com/#subscription%2Ffeed%2Fhttp%3A%2F%2Foffbeatfamilies.com%2Ffeed
EMPIRE:
http://cloud.feedly.com/#subscription%2Ffeed%2Fhttp%3A%2F%2Foffbeatempire.com%2Ffeed
I will say that when I went to those URLs, I was shown as NOT subscribed… I’m not sure if that’s caused by our server weirdness, or something with Feedly, but it was easy to resubscribe.
UG. I’m so sorry about this. I honestly have no idea what’s going on. 🙁
Yep, I checked my Feedly list and was indeed unsubscribed from my Offbeat sites except for the Empire. I had just started to notice my Feed was “quiet” when I saw the post at OBE about the sites being down on Sept 6th or so. I wonder how many other subscribers are unaware that they somehow got unsubscribed? I wonder if you can tap into those stats, to see the number of subscribers from before and after the meltdown. Worth noting, that it looks like I got unsubscribed days before the meltdown happened, around Aug 26th or so.
First: I realized I was wrong: I was actually still subscribed to the feeds. D’oh. My confusion.
Looking at my referral stats, the numbers of people coming from Feedly has stayed totally consistent for the past 3 months, so I’m hoping your experience is an isolated situation.
Sadly, very few of my readers use RSS (I *SO* wish more did). Like you, RSS followers tend to be more invested than you average reader… so I’m just going to have to trust that folks are finding their way to the RSS feeds when/if things go wonky.
That said, as far as I can tell from my stats, there’s been no real drop in folks coming from Feedly, which is good.
You melted that old server with sheer awesomeness!
I had a few tabs open in OBB when the site went down, so I kept them open, figuring I’d reload once I heard that the site was back. But now they seem to have vanished – it was “Non Legal Commitment Ceremonies”, “Amazing Wedding Venues”, and “Photo Booth Drink Sign”. Just figured I’d ask, ’cause I was really curious about those amazing wedding venues!
Nice sleuthing! Those were posts that were scheduled to go up yesterday, that we decided to hold on to until next week. You’ll see them again Monday and Tuesday!
I don’t know if this had anything to do with your old servers, but often I would check all three websites in a row and open up a bunch of tabs from them to read through, and often (especially on OBH) one or several of the pages would say that the server was not responding, and then if I refreshed any of the other pages in the Empire that had already successfully loaded it would say that the server wasn’t responding for them either. It would usually only work itself out if I went back after an hour or so. Dunno if that will help, but I always wondered why the Empire websites would do this when no other ones I had encountered ever did the same thing.
Kudos to the techs for getting it all back up so quick really after a mega outage. New shiny server – seems new and shiny! and I threw a donation to the Whisky fund.
I love the OBE – It’s my go to place for happy stuff when RL (Real life) is being too much.
Oh oh! Is the change that the slideshow thing-a-mabob has gone? 🙂
*sits quietly and hopes for a gold star*
Nope, although… does this mean you’re not seeing the post slider on the homepages? I’m seeing it on all our sites. Clear your cache and see if it appears?
Oh….!
No I can’t see it, although I am on my phone (htc using chrome browser). Would that make the difference?
Yep. On mobile widths, the slider is supressed. 🙂
It’s kind of awesome that no one’s noticed the difference yet… it feels HUGE to me, but obviously I look at the sites differently. 🙂
And I was really hoping for a gold star… Maybe next time!
It was the very first thing I noticed!!! Then I read the post, saw the question and got all excited. ‘Till you already had the answer up 🙁
I know I’m commenting after you gave us the answer, but can I still get a gold star?
aren’t there usually links on the sidebar for suggested pages? or something? i feel like something is missing from the sidebar…
i was so lost without OBE to browse during work the other day. i’m glad things are starting to get back on track!
The sidebar used to have tag clouds, which we got rid of a few weeks ago at the advice of a UX consultant who was like TOO MANY LINKED WORDS JEEZ TAGCLOUD WHAT IS THIS 2006?
is it the shape of the tabs? as in the blog, vendor guide etc tab? were they always so tab shapped? man its killin me i cant figure it out…. i thought it was the lack…. or the drop down menus??? ooooh wait am i missing a the ‘decor porn, wedding porn’ frop down menus?
nope scrap that i just found those
wait i think i got it…. the facebook, twitter, more options buttons on the left hand side…. i dont remember seeing those before
Nope, those share buttons were part of Smart Layers, which we added last month: https://offbeatempire.com/2013/08/addthis-smart-layers 🙂
It would be easier to spot the change if I could see them side-by-side, but here’s my shot at it anyway. Did the very top bar (with links to the other sites) used to be static (i.e. stayed at the top of the browser window when you scrolled down the page)?
Nope, although that’d be clever!
The “recommended for you” posts are at the bottom of the page? I fee like they used to be on the side bar? Curiouser and curiouser!
Nope! Those were also part of the Smart Layers addition last month: https://offbeatempire.com/2013/08/addthis-smart-layers 🙂
Hee hee. Oh you guyyyys. This is stupidly more fun than it should be. 🙂
What’s REALLY interesting is that we’ve had a couple people complain about how we had things, saying they wished we did it the way we’re now doing it… so some people feel strongly about the issue.
Is it THIS? Because was that like <3 before? And it's like http://soblue for links
Nope! This has always been THIS! on the blogs, and a heart on the Tribe.
Don’t ask why. 🙂
… Why? 😉
OH YOU.
The most honest answer is that I can’t quite remember. But we talked about it a lot. I think it had something to do with wanting Tribesmaids to be able to offer support (heart!) without always agreeing (this!). Plus, it gives one more little visual indication that you’re on the Tribe, not the blog… and some Tribesmaids get confused about what’s what on the site.
Well there you go… I <3 THIS!