Forum:Announcements/Hyper-aggressive AI LLM crawlers
Forums → Official announcements → Announcements/Hyper-aggressive AI LLM crawlers
The main OGF server and wiki server have both been under increasing load due to hyper-aggressive AI LLM crawlers. This impacts site performance, and particularly on the wiki server has resulted in severe outages. Adding additional server resource only offers a partial respite, as the requests then increase yet again.
These disruptive requests are extremely hard to mitigate. They ignore robots.txt instructions, operate from an immense range of IP addresses and use UserAgent strings which randomise across valid agents used by real users.
As a mitigation we are experimenting with HTTP Authentication. If you see such a request then enter ogf as the username and opengeofiction as the password. This may be implemented sporadically.