Chinese Spiders

[russellw]

Good afternoon.

No, I'm not posting about some exotic 8-legged creature but about the web crawlers that have been hammering this site for the last 5 days or so.

These crawlers are ignoring the normal directives that robots are supposed to follow and thus we have several hundred of them crawling the site constantly.

The impact is that the server CPU's get maxxed out at 100% and overall performance slows to a (no pun intended) crawl.

They can normally be kept at bay by blocking them at the server level but for some reason this doesn't appear to be stopping this lot of Huawei crawlers.

There have been problems reported around the world from them so we aren't the only ones being impacted as Huawei try to get their search engine up to the standard of Google as part of their ongoing war with each other.

We are working with out ISP to find a solution.