Delete a Message
As usual Google bot & Bing bot are in the top of the list.
I see that the Google bot competely ignores the rare set in robots.txt, they fetch pages at a too high rate (much more than a normal user browsing the site). They used to have an option to temporary reduce the rate, but it does not seem to work
Yesterday we also had that one joining the party:
https://dataforseo.com/dataforseo-bot
As well as the Amazon bot
The first IP address of a (probably) real user is at the 10th place in the list...
I see that the Google bot competely ignores the rare set in robots.txt, they fetch pages at a too high rate (much more than a normal user browsing the site). They used to have an option to temporary reduce the rate, but it does not seem to work
Yesterday we also had that one joining the party:
https://dataforseo.com/dataforseo-bot
As well as the Amazon bot
The first IP address of a (probably) real user is at the 10th place in the list...