The article doesn't mention it, but it entails a geo-block, not a worldwide block. Github will probably honor the ineffective geo-block.
Because of the Streisand Effect, though, the Indian government made the "problem" far worse for themselves. These mesh networks will grow much faster now, worldwide. They will have even less control over people's communications.
This might have something to do with lemmy.kde.social no longer existing. It gives a 502, Bad Gateway error. This server hasn't had communication with it in over 2 months and has finally flagged it as "gone forever."
It's possible that under this circumstance, PieFed, rather than providing an error when you try to post to an unreachable community, just posts to a different community.
YouTube requires a valid Referer header to allow embedded playback, and if Nginx is stripping it out, like mine did, the videos throw an error and refuse to play.
I fixed this by updating the referrer-Policy header. I updated the server {} block with this:
I've always believed, until this week, that if I posted something and it got upvotes, it was because people liked that I posted it. I've learned this week that because of some, it's become a much less meaningful indicator. It's been co-opted for unrelated purposes. I no longer trust it has any meaning. This is a letdown. Let's add it then to the list of meaningless metrics like lines of code, story points, and StatCounter statistics.
If it remains an admin choice, I will keep it enabled for the two reasons that the behavior makes vote counts meaningless, which is unfair to the community, and that these empty votes add a ton of unnecessary traffic for every instance. I wonder what percentage of my server is used just to process meaningless votes. Each vote sends me a request to update the database. Each meaningless vote here requires me to send out a ton of requests to other servers. Why should I pay for this? Why should users see lower performance because of this behavior?
In the past 24 hours, only 13 requests got rate-limited by the firewall. No rate limits in the last 24 hours from the server itself. The 13 was part of a burst of requests coming from lemmy.world. But just 13 the whole day. Unsure where else the 429 error could have come from. Seems unlikely it was from feddit.online.
Rate limiting is possible. Currently, 1/3 of all the traffic coming in (about 200K requests/day) are bots. Rate limits were necessary. How often do you vote within a minute?
Do you do a lot of voting? Version 7 puts a limit on how much voting can be done in a day. I don't know the limit or the details. If your quota bar is solid red, it's the reason. Let me know.
There are two answers to this depending on what the reason is for asking.
If you are asking because you are concerned about scrapers reading your posts and violating your privacy and your rights, then understand that even if an instance is 100% effective at blocking them, the post is sent all over the place in clear text anyway. It doesn't matter for them which of the federated servers your post is read from. They will read your post many times over. For this case, then, there is little incentive for a server owner to block bots if it's just to protect your posts from ingestion.
If you are asking because you are concerned about scrapers sucking the life out of a server because there are multiple different AI companies trying to read every single post in the database multiple times over for training, which ends up causing gateway timeout errors and poor performance, then admins, for this reason, should take action.
On my PieFed server, feddit.online, as of yesterday, the firewall discarded 99K requests it deemed were for AI scraping while processing the remaining 300K requests. Those 99K requests would have been expensive requests, not just upvotes and such, but requests asking for huge amounts of text, and so the impact on the server and infrastructure would have been much more than a 25% tax on the system.
And if the bots realize your server is not well protected, it gets worse. 3 months ago I peaked at 1.2 million requests in one day, of which over 700K were AI bots. Now it's down to consistently under 100K from bots because many of them have given up, I like to believe.
I upgraded the server to 1.7 the day it was released. It was released yesterday, July 3rd. This morning I updated it to 1.7.1, which was released today. Your post was made before the upgrade to 1.7
Yes, Cloudflare has tools to detect bot traffic, which include curl and similar utilities. It absolutely stands in front of the server.
But if the correct URL cannot be opened by a mobile client, then there's a problem. Then I'd need the details.
The firewall is trying to only allow traffic from either web browsers, other ActivityPub servers, or mobile applications. It's designed to block traffic from anywhere else.
I just checked. In the past 24 hours, the firewall has dumped 98K requests from AI bots that tried really hard to look like normal traffic and not like AI bots. These bots did a lot more than just set a user agent to try to look like real users, but it didn't help them.
I tried recreating the issue using Summit on an Android while logged into piefed.social and I had no issues reading all your posts or seeing all your pictures. Some things to note:
When Cloudflare tried to determine if it was a bot request, the process stopped when a 404 was returned by the server. Hence, Cloudflare's attempt to prove the request was not from a bot failed due to this error, so a 403 was returned by Cloudflare.
Curls will never work except to prove the firewall is working. Curl requests quack like a bot, they look like a bot, and they smell like a bot and are therefore identified as a bot and not allowed.
Things work when the requests come from real web browsers, and they should work when the requests come from a real phone app. Was gemelen using either?
There are no restrictions on Uruguay.
If gemelen is trying to read the post through a real web browser or a real mobile app, the request should work. If he is using a real web browser or phone app, and using the correct URL, then please give me the details so I can check on why the requests were blocked.
No problems whatsoever for me
And thank you VERY MUCH for creating something so awesome! I love using it.
The article doesn't mention it, but it entails a geo-block, not a worldwide block. Github will probably honor the ineffective geo-block.
Because of the Streisand Effect, though, the Indian government made the "problem" far worse for themselves. These mesh networks will grow much faster now, worldwide. They will have even less control over people's communications.
This might have something to do with lemmy.kde.social no longer existing. It gives a 502, Bad Gateway error. This server hasn't had communication with it in over 2 months and has finally flagged it as "gone forever."
It's possible that under this circumstance, PieFed, rather than providing an error when you try to post to an unreachable community, just posts to a different community.
I added a couple of devs.
@rimu@piefed.social @wjs018@piefed.social
YouTube requires a valid Referer header to allow embedded playback, and if Nginx is stripping it out, like mine did, the videos throw an error and refuse to play.
I fixed this by updating the referrer-Policy header. I updated the server {} block with this:
Does anyone know why it's been put on serverHold?
I got banned from Reddit for mentioning my Mastodon server in a post. You have a lot of company.
I've always believed, until this week, that if I posted something and it got upvotes, it was because people liked that I posted it. I've learned this week that because of some, it's become a much less meaningful indicator. It's been co-opted for unrelated purposes. I no longer trust it has any meaning. This is a letdown. Let's add it then to the list of meaningless metrics like lines of code, story points, and StatCounter statistics.
If it remains an admin choice, I will keep it enabled for the two reasons that the behavior makes vote counts meaningless, which is unfair to the community, and that these empty votes add a ton of unnecessary traffic for every instance. I wonder what percentage of my server is used just to process meaningless votes. Each vote sends me a request to update the database. Each meaningless vote here requires me to send out a ton of requests to other servers. Why should I pay for this? Why should users see lower performance because of this behavior?
Please leave the setting, Rimu.
And I completely like the proposal.
In the past 24 hours, only 13 requests got rate-limited by the firewall. No rate limits in the last 24 hours from the server itself. The 13 was part of a burst of requests coming from lemmy.world. But just 13 the whole day. Unsure where else the 429 error could have come from. Seems unlikely it was from feddit.online.
Rate limiting is possible. Currently, 1/3 of all the traffic coming in (about 200K requests/day) are bots. Rate limits were necessary. How often do you vote within a minute?
Do you do a lot of voting? Version 7 puts a limit on how much voting can be done in a day. I don't know the limit or the details. If your quota bar is solid red, it's the reason. Let me know.
There are two answers to this depending on what the reason is for asking.
If you are asking because you are concerned about scrapers reading your posts and violating your privacy and your rights, then understand that even if an instance is 100% effective at blocking them, the post is sent all over the place in clear text anyway. It doesn't matter for them which of the federated servers your post is read from. They will read your post many times over. For this case, then, there is little incentive for a server owner to block bots if it's just to protect your posts from ingestion.
If you are asking because you are concerned about scrapers sucking the life out of a server because there are multiple different AI companies trying to read every single post in the database multiple times over for training, which ends up causing gateway timeout errors and poor performance, then admins, for this reason, should take action.
On my PieFed server, feddit.online, as of yesterday, the firewall discarded 99K requests it deemed were for AI scraping while processing the remaining 300K requests. Those 99K requests would have been expensive requests, not just upvotes and such, but requests asking for huge amounts of text, and so the impact on the server and infrastructure would have been much more than a 25% tax on the system.
And if the bots realize your server is not well protected, it gets worse. 3 months ago I peaked at 1.2 million requests in one day, of which over 700K were AI bots. Now it's down to consistently under 100K from bots because many of them have given up, I like to believe.
I upgraded the server to 1.7 the day it was released. It was released yesterday, July 3rd. This morning I updated it to 1.7.1, which was released today. Your post was made before the upgrade to 1.7
Statcounter is worse than a guess. There are numerous articles, but here's a good one: Statcounter claims millions of people stopped using iphones and switched to Windows 7
I made no changes. You might want to raise this in the PieFed help community.
Yes, Cloudflare has tools to detect bot traffic, which include curl and similar utilities. It absolutely stands in front of the server.
But if the correct URL cannot be opened by a mobile client, then there's a problem. Then I'd need the details.
The firewall is trying to only allow traffic from either web browsers, other ActivityPub servers, or mobile applications. It's designed to block traffic from anywhere else.
No, the agent string is not enough.
I just checked. In the past 24 hours, the firewall has dumped 98K requests from AI bots that tried really hard to look like normal traffic and not like AI bots. These bots did a lot more than just set a user agent to try to look like real users, but it didn't help them.
I tried recreating the issue using Summit on an Android while logged into piefed.social and I had no issues reading all your posts or seeing all your pictures. Some things to note:
When Cloudflare tried to determine if it was a bot request, the process stopped when a 404 was returned by the server. Hence, Cloudflare's attempt to prove the request was not from a bot failed due to this error, so a 403 was returned by Cloudflare.
Curls will never work except to prove the firewall is working. Curl requests quack like a bot, they look like a bot, and they smell like a bot and are therefore identified as a bot and not allowed.
Things work when the requests come from real web browsers, and they should work when the requests come from a real phone app. Was gemelen using either?
There are no restrictions on Uruguay.
If gemelen is trying to read the post through a real web browser or a real mobile app, the request should work. If he is using a real web browser or phone app, and using the correct URL, then please give me the details so I can check on why the requests were blocked.
Lemmy is t-i-m-eless.