Not only wikis sadly. Anything that has public facing deep links that trigger extensive database operations are being hammered by these bots and few servers can take the load.
In coordination with careful rate-limiting, it’s been a dramatic improvement.
The worst of the bots don’t advertise their User Agent (or worse, attempt to present they’re a normal user making 100s of requests a second) but there’s lots of low hanging fruit.
On the plus side, this isn't a problem with AI, this is a problem with AI companies having more investment money than they know what to do with. The moment the hype fades and they don't want to hemmorage money scraping every wiki on the internet thousands of times per day, this traffic will go back to a far more sane amount.
Maybe we should have 2 factor authentication to read wikis. A person would have to come here and seek a public key from other people. If you can get 2 Public keys by proving you are human, then you get a month of wiki access unless you keep proving you are human or if you prove to be AI then you're totally banned unless you get a public key in real person at the local Walmart or Safeway.
7 Comments
poVoq@slrpnk.net · 22 pts · 85d
Not only wikis sadly. Anything that has public facing deep links that trigger extensive database operations are being hammered by these bots and few servers can take the load.
thatsnothowyoudoit@lemmy.ca · 12 pts · 85d
We use NGINX’s 444 response A LOT.
In coordination with careful rate-limiting, it’s been a dramatic improvement.
The worst of the bots don’t advertise their User Agent (or worse, attempt to present they’re a normal user making 100s of requests a second) but there’s lots of low hanging fruit.
grrgyle@slrpnk.net · 6 pts · 85d
Hmm interesting. I wasn't aware of this one
Tiresia@slrpnk.net · 8 pts · 85d
On the plus side, this isn't a problem with AI, this is a problem with AI companies having more investment money than they know what to do with. The moment the hype fades and they don't want to hemmorage money scraping every wiki on the internet thousands of times per day, this traffic will go back to a far more sane amount.
rasterweb@fedia.io · 3 pts · 83d
So we should make more wikis filled with poison data… got it! ;)
poVoq@slrpnk.net · 3 pts · 85d
It will probably go down, but the process itself is kind of unavoidable for training LLMs, so I doubt things will go back to how they were before.
altphoto@lemmy.today · 1 pts · 65d
Maybe we should have 2 factor authentication to read wikis. A person would have to come here and seek a public key from other people. If you can get 2 Public keys by proving you are human, then you get a month of wiki access unless you keep proving you are human or if you prove to be AI then you're totally banned unless you get a public key in real person at the local Walmart or Safeway.