Don’t ban the bots
I do a lot of DDoS related research online, which results in a lot of DDoS protection related spam/offers. A trend I have seen gaining popularity lately is “ban the bots”.
These emails contain a lot of emotionally charged language trying to persuade the reader that bots are destroying the internet, wasting your bandwidth and pillaging your website (and how for a modest monthly fee they can keep the digital invaders at bay). I couldn’t disagree more. For the most part I like bots. Bots save me a ton of work and allow me to the focus on tasks that are meaningful to me. The only reason that search engines, hotel booking sites, and social media sites operate so successfully (or at all) is because of bots.
These advertisements do acknowledge there are some good bots out there, while stressing the need to block the bad bots. I thought I’d pull some numbers from traffic running through our system. I was pleasantly surprised, as a DDoS protection service I was expecting to see more malicious bots than legitimate but what I found was 85% of the bot traffic is classified as good: SES (which stands for Search Engine Spiders, but is a general list of the known good bots) which we don’t want to block, and XSE which contains alternate Spiders and bots that while legitimate can cause impact on some websites.
The other 15% of traffic is from hosting companies, ISPs, and commercial traffic from unknown bots. This traffic is not automatically bad, but hidden somewhere in there are the malicious bots and scrapers which we do want to block. This is where the philosophy “ban the bots” makes things more complicated than it needs to be, because while it is a trivial matter to find and locate bots, it focuses you on the actor not the action. Don’t ban the bots, ban the malicious actions. If you design your web security to defend against malicious actions it shouldn’t matter whether they are from bots or not. At DOSarrest this is what we do, we create
special features to focus on the malicious bot traffic and apply them to customer configurations and leave the good bots alone.
In fact, I’ll go one step further: don’t ban the bots, help the bots. Because while I disagree with the conclusion the facts are not wrong, bots do consume more than a trivial amount of resources. By helping the bots find the content they are looking for you can reduce the impact on your site and possible improve your overall ranking.
Security Solutions Architect
DOSarrest Internet Security