"Server slow" but perhaps different reasons.  The number of spiders crawling our site has been creeping up - to some extent as expected once we moved from "http" to "https".   As I see individual spiders adding a significant load, I at taking a look at those that come onto my radar and asking (or forcing) some of the more noisy ones not to index our content. 

In terms of web searches, 19 out of 20 in the UK use Google and the Googlebot is allowed Coffee Shop access just as any human guest might be (so no indexing of things like Frequent posters) but others such as Yandex and Petal which are not widely used in the UK are turned away.  Ironically, Google does not put a big load at all on when indexing - I see it, but it's not effecting performance like (for example) the Petal crawler was earlier today.

It's not just web search indexers that crawl sites ... there are a variety of other companies indexing and selling data to their customers or making use of it within products - from plagiarism identifiers to AI feeds and Search Engine Optimisation tool and link reporters.  In some cases they may do us a bit of good, but as we're not driven by sales volume and income here, and we're very much UK based when they tend to index worldwide, that good if it's there tends to be tangential. 

It's a bit of an ongoing game - others play it too and have things like "Captcha" - "are you really human" test which we could do if we need to.   We do have some logic that spots aggressively or characteristically automated visiting without the need for users to tell us which boxes have motorcycles or stairs (or GWR class 158s!) in them from time to time.   Please let me know if you get a "you are not really human" type message more than very rarely!

 

OK - dealt with ... the heavy black line is today's worker server load.  Looks dramatic - there was (perhaps) a handful of proper requests that may not have been answered but the server did not fall over.



If you wonder how I find out there's a problem like this, the server sends me a message 

Now in Vilnius

 
Noting that our server is suffering a denial of service attack - 44 requests PER SECOND.   Dealing with it as best I can from where I am (I'm on a train - in Lithuania!)