Posted by
I’ve gotten a half-dozen or so emails about this issue over the past few months, so I figure there are probably lots more of you experiencing it and NOT emailing… so I better address it!
Our web servers have some anti-spambot tools running at all times, watching for computers accessing the sites in aggressive ways. See, spambots will sometimes hit Offbeat Empire webpages hundreds of times in a minute, and when several of them do this at the same time, it can threaten to take down the entire server.
Once our anti-spambot tool thinks it’s identified a spambot, it blocks that IP address from accessing the server at all for three days. Basically, a spammer hits the sites too many times, too hard, too fast, and they get locked out.
But some of you readers like to hit it hard and fast too. And sometimes our anti-spambot tools get confuse and misidentify legit readers as spambots, and dark them out. The pages just stop loading. There’s no error, no messaging… there’s just NOTHING. You’ve been darked out.
Now if this happens to you, don’t feel bad! Even members of our staff have accidentally been locked out — it happens. Here’s what to do if you accidentally get darked out:
- Go to downforeveryone.com to make sure it’s just you. The Offbeat Empire is pretty stable these days, but the sites do still have hiccups.
- Once you’ve confirmed that it’s just you, go to whatismyip.com and copy your IP address. It’ll look sorta like this: 12.34.567.890
- Email me letting me know that you think you’ve been blocked out, providing your IP address.
It’s super quick and easy to get you reinstated quickly… otherwise you have to wait three days for our server to let you back in, and we all know y’all can’t handle three days without your Offbeat Empire stories!
I’ve only been getting a fraction of new post emails for well over a year now (I didn’t even get the email when my wedding was published). I’m not super active over here anymore, but I do like getting notified of new posts. I’ll still try your steps anyway even though I’m pretty sure it’s not because I’m hitting the pages too fast and it thinks I’m a spambot.
The new post emails are totally unrelated to our server’s spam blockers. All our new post emails are sent through Mailchimp, and I can see that they’re going out the same way they always have. I suspect spam filtering — whitelist [email protected], check your spam folders, and then perhaps talk to your mail provider?
This is the joy of using Mailchimp: they are suuuuper reliable, and I can see from my Mailchimp stats that everything is going out as it should and nothing has changed over there.
So while this is a totally separate issue from getting darked out, let’s get it figured for you! 🙂
Kristin, I just searched the mailing lists for the email address you used with your comment here (a hotmail.com account) … and I’m not seeing the address as subscribed to any of the mailing lists. Did you use a different email address?
I can manually resubscribe you if you’d like. Go ahead and email me: https://offbeatempire.com/contact
I get the occasional new post email but not very many. Maybe once a week or every two weeks. I subscribed using the same email
Can you email me so we can troubleshoot? https://offbeatempire.com/contact
I want to see which emails you’re receiving so we can troubleshoot why you’re not receiving all of them.
WARNING : Extreme geekiness ahead.
Given that anybody’s IP can change at any time and given that ISPs can use 1 IP for multiple machines, how does your anti-spambot software decide how to block? It seems like in order to guarantee a block you’d have to take the whole subnet ( or more ) but that would be very harsh. But even if you limit yourself to 1 IP you might wind up blocking an entire ISP’s worth of users. Does it actually depend on the MAC address? ( And even if it did I’m not sure that would suffice since I’m not sure how ISP’s obscure that. )
I’m also curious about how often a user would out-click a ‘bot. Usually software wins but in this case the ‘bot is unnaturally gated in that it has to wait for the web server to respond completely and the impatient user, who clicked again before the page finished loading, is not. But the ‘bots actions are more predictable. It’s going to go through your site methodically in a way no user would do. That would seem a better indicator than “speed of click” unfortunately while the anti-spambot software is determining this, half your site might have been downloaded.
So can you tell which error is actually occurring? Can you tell if a user has been mistakenly identified as a spambot because his or her actions matched the heuristics or if the user has been banned because a real spambot on another machine was operating on that IP as well ?
Honestly it’s not that sophisticated. It uses iptables to block by IP based on some rules around traffic patterns we came up with by watching some massive attacks in realtime. The basic formula is X hits in X minutes to X urls. Most of the wordpress-specific bots we’ve seen are targeting the same spots (login, comment forms on post pages, a few other known weak points for exploits) en masse over hours. They don’t wait for a response, they just spray requests all over the server hoping to inject content somewhere.
A fair amount of bot traffic still gets through – I can see it in the logs – but it’s not enough to cause major issues. I don’t care if a bot wants to hit me every 10 minutes for the next year, but I do care if it brings 100 friends to hit us every 3 seconds for the next 6 hours.
The attacks that are big enough to cause us problems are usually from just a handful of IPs, so there’s no need to block the entire subnet. But because both the block and the expiration are automatic it doesn’t really matter how many IPs are involved or where they’re coming from. They all get blocked just the same.
In your basic HTTP request it is not possible to get the client’s MAC address. I believe there are exceptions to this if you’re on the same local ethernet, but unless you like hanging out in datacenters this isn’t going to happen. 😉
Unfortunately we really can only identify false positives when readers pipe up that they can’t access the sites. After this post went live we adjusted the lockout time that those folks wouldn’t be in the dark so long (3 days is a little excessive). There are more sophisticated ways of identifying real vs bot traffic, but since I’m the only tech person on staff we have to make choices about what to prioritize development-wise.
I hope that answers some questions!
Innnnnerestin….
Thanks for taking the time to fill me in.
I got locked out for around three days last week. My IP is in Indonesia and I imagine that there are more than a few spambots operating on my same ISP, so… Now it all makes sense!