Also, while improving server performance and failure modes under load would help, the “badly-written, aggressive scraper bots that ignore robots.txt and are given loads of network resources clobbering servers” is something that affects many, many different Web servers out there. This isn’t a Lemmy problem, nor even just social media problem. It’s a Web problem.
100%. Everyone gets this problem and then they put their site behind cloudflare. But what happens when the entire Internet is behind an american company who can decide which sites it likes and which it doesnt?
Same thing for the Threadiverse. The Threadiverse will let you set up a Threadiverse instance and subscribe to everything, efficiently feed all the posts and comments you want to your instance, the moment they come in. In nice, machine-readable form, rather than in something intended for humans that you have to scrape and post-process. But…it takes more dev effort to set up something specific to the Threadiverse than to just treat it like another website.
Yeah, the web was intended to be this universal format that would always work on any platform and any operating system. But the downside is clear today. Some estimates say that we already have much more bot activity than human activity on the internet. And we have AI creating content for the last few years...
I could but there are so many requests, like 30-50 per second, and even if I could cache the login page, I would still spend the bandwidth to send it back to each of those bots. So because of that, and because its not even the main lemmy.today site, I figured its OK to just return 403.
Server load has decreased to almost nothing on old.lemmy.today. During the years we have been online, more and more bots are scraping and now its just ridicolous. :)
No, I checked and we do require email. I wasnt sure what was set there but yep, its required. But just put any email address there if you dont want to use a real one. You just wont get an email with approval or denyal after account creation, so you have to come back later and try to log in at some point.
It's fine to not give an email address. It's just for the email to let you know that you were approved or denied. So if you put something random there and check if you can login later, it's all good.
I think it's even optional to give it... You can try registering without. :)
Lemmy federation has been known to be buggy in certain releases before, and I really dont know what to do about it except try to keep up to date about bugs. I guess its a complex system underneath. From what ive read, its a queue of federation work that is supposed to take place, and stay in the queue until its successful. But I wish there was a good way to see the status of it.
I have been looking in db tables and log files before to figure out the current status and it works, but its not so convenient.
Yeah the API should be exempt. Will look into this a bit more in the coming days but wanted to see how well Anubis works on old.lemmy.today first for a while. We probably end up using it on lemmy.today as well. Just didnt want to make a big bang change everywhere on day 1. :)
It sits between your web browser and old.lemmy.today and detects if you are a bot. If you are, your request doesn't pass through Anubis. It's Gandalf saying "you shall not pass".
So it will hopefully cut down on bots scraping the instance. Or it will piss them off even more. :)
I added Anubis for https://old.lemmy.today/ and will see if it helps the situation. Bots are hammering it pretty hard but they are not getting any lemmy content at least. Thank you for your advice. :)
Ok now its working and seemingly blocking bots, but we will see over time how well this works. Like you said, im sure its possible to bypass but maybe it improves the situation somewhat still.
They pass dystopian chat control with one hand and does this with the other.
My arch system is so stable it's boring. Year after year, just runs. Doesn't slow down, doesn't install crapware.
So I dont know. I'm happy with it.
100%. Everyone gets this problem and then they put their site behind cloudflare. But what happens when the entire Internet is behind an american company who can decide which sites it likes and which it doesnt?
Yeah, the web was intended to be this universal format that would always work on any platform and any operating system. But the downside is clear today. Some estimates say that we already have much more bot activity than human activity on the internet. And we have AI creating content for the last few years...
I totally agree with you actually. Maybe there will be some better solution going forward.
@tal@lemmy.today @PapaSkwat@lemmy.today This should be fixed now. It was caused by incorrect caching of the posts. :)
Absolutely. :)
Hi,
I could actually reproduce this myself now:
https://lemmy.today/post/57605044
This link only works sometimes for me too. I have no idea why so will dig into the server logs. :)
I could but there are so many requests, like 30-50 per second, and even if I could cache the login page, I would still spend the bandwidth to send it back to each of those bots. So because of that, and because its not even the main lemmy.today site, I figured its OK to just return 403.
Server load has decreased to almost nothing on old.lemmy.today. During the years we have been online, more and more bots are scraping and now its just ridicolous. :)
No, I checked and we do require email. I wasnt sure what was set there but yep, its required. But just put any email address there if you dont want to use a real one. You just wont get an email with approval or denyal after account creation, so you have to come back later and try to log in at some point.
It's fine to not give an email address. It's just for the email to let you know that you were approved or denied. So if you put something random there and check if you can login later, it's all good.
I think it's even optional to give it... You can try registering without. :)
Make a PR. :)
Ive seen this before too but never thought about what it was. :)
Lemmy federation has been known to be buggy in certain releases before, and I really dont know what to do about it except try to keep up to date about bugs. I guess its a complex system underneath. From what ive read, its a queue of federation work that is supposed to take place, and stay in the queue until its successful. But I wish there was a good way to see the status of it.
I have been looking in db tables and log files before to figure out the current status and it works, but its not so convenient.
Yeah the API should be exempt. Will look into this a bit more in the coming days but wanted to see how well Anubis works on old.lemmy.today first for a while. We probably end up using it on lemmy.today as well. Just didnt want to make a big bang change everywhere on day 1. :)
Yeah, this is just a start. :)
Hope so!
It sits between your web browser and old.lemmy.today and detects if you are a bot. If you are, your request doesn't pass through Anubis. It's Gandalf saying "you shall not pass".
So it will hopefully cut down on bots scraping the instance. Or it will piss them off even more. :)
I added Anubis for https://old.lemmy.today/ and will see if it helps the situation. Bots are hammering it pretty hard but they are not getting any lemmy content at least. Thank you for your advice. :)
Now it should work well again. I put it behind anubis and we will see if it improves the situation with the bots.
Ok now its working and seemingly blocking bots, but we will see over time how well this works. Like you said, im sure its possible to bypass but maybe it improves the situation somewhat still.