Hi,
I had a couple of people mention they were getting an error trying to access a link I posted in c/fountainpens. One of the users (@gemelen@mammut.moe) created a traffic dump of trying to access the link using curl. He posted his results on this Opensuse paste bin.
He added this:
I guess Cloudflare prevents some web clients (like embedded viewer for a Mastodon mobile app) and some networks (like mine Uruguayan Internet provider) - I could open the feddit.online and see your post there in a desktop browser, but being forbidden to access from a mobile or via curl.
George
12 Comments
feddit-admin@feddit.online · 1 pts · 43d
When I suddenly began receiving over 750K requests per day from AI bots, I did significantly tighten firewall rules so the server could get back online (it was throwing gateway timeouts). This issue is probably collateral damage.
The firewall does not block any ISPs in Uruguay, so I don't believe Uruguay is a factor.
There are several new layers of firewall rules. One of them is blocking the request.
The information I need to adjust the firewall may be in the paste bin post (thank you for this!). But if there is anything more you can tell me about what is trying to read the message, it would be helpful. Which phone app, for example?
I'll have more time to look at this tomorrow. I may have more questions. Hopefully I can reproduce the issue.
Thanks for reporting this!
Jerry@feddit.online · 1 pts · 43d
Jerry@feddit.online · 1 pts · 43d
I tried recreating the issue using Summit on an Android while logged into piefed.social and I had no issues reading all your posts or seeing all your pictures. Some things to note:
When Cloudflare tried to determine if it was a bot request, the process stopped when a 404 was returned by the server. Hence, Cloudflare's attempt to prove the request was not from a bot failed due to this error, so a 403 was returned by Cloudflare.
Curls will never work except to prove the firewall is working. Curl requests quack like a bot, they look like a bot, and they smell like a bot and are therefore identified as a bot and not allowed.
Things work when the requests come from real web browsers, and they should work when the requests come from a real phone app. Was gemelen using either?
There are no restrictions on Uruguay.
If gemelen is trying to read the post through a real web browser or a real mobile app, the request should work. If he is using a real web browser or phone app, and using the correct URL, then please give me the details so I can check on why the requests were blocked.
gemelen@mammut.moe · 1 pts · 43d
@Jerry
To clear this out: curl _may_ work even in that particular case with a provided user-agent and other things, if necessary. But the most importantly, it demonstrated that the Cloudflare is in front of your resource.
Originally, the URL that was shared by @Unattributed is "https://feddit.online/c/fountainpens/p/1803832/why-fountain-pens", so it was attempted to visit from the mobile (either Tusky or Pachli app - Mastodon clients for Android, and from mobile Vivaldi).
Thus I (and accidentally other person) replied that it's unreachable through these methods. On the other hand, it could be opened from a desktop browser (desktop version of Vivaldi specifically).
Where that URL came? Likely, copied from the browser's address bar, I would assume. But that's a question to @Unattributed
Jerry@feddit.online · 1 pts · 43d
Yes, Cloudflare has tools to detect bot traffic, which include curl and similar utilities. It absolutely stands in front of the server.
But if the correct URL cannot be opened by a mobile client, then there's a problem. Then I'd need the details.
The firewall is trying to only allow traffic from either web browsers, other ActivityPub servers, or mobile applications. It's designed to block traffic from anywhere else.
Unattributed@feddit.online · 1 pts · 42d
I think I see the issue… Looking at the post from mastodon.social this is what comes through:
@gemelen@mammut.moe was seeing the post as above: from Mastodon... Apparently when the message was federated to the rest the FediVerse the generated URL is wrong. The
@lemmy.worldportion was removed. I think this is a bug that Rimu will need to look at...Unattributed@feddit.online · 1 pts · 42d
Actually, there is an issue open on this already…
https://codeberg.org/rimu/pyfedi/issues/1831
Okay, that was a duplicate… The original issue was filed here: https://codeberg.org/rimu/pyfedi/issues/1809
Rimu has acknowledged it.
Unattributed@feddit.online · 1 pts · 42d
When did we upgrade to 1.7? Rimu says this issue was fixed in 1.7, but I can't tell if the original post was made before or after the upgrade...
Jerry@feddit.online · 1 pts · 42d
I upgraded the server to 1.7 the day it was released. It was released yesterday, July 3rd. This morning I updated it to 1.7.1, which was released today. Your post was made before the upgrade to 1.7
Unattributed@feddit.online · 1 pts · 42d
So, that should have resolved the issue, but it's probably best to do another post in a group with the domain as part of the name...
Unattributed@feddit.online · 1 pts · 43d
I've just asked gemelen if he can join us to answer your questions.
I was thinking about what you said regarding curl (originally in your deleted post): would setting the user agent have had any effect? I seem to recall having done that in the past for some specific script I wrote (but that was over 15 years ago, so my memory may be shot).
Makes sense that if the URL was wrong in the first place, then the whole thing would fail with a 404 or 403.
gemelen did say that when he tried it from his desktop it worked — hence the reason he suspected it was an issue between cloudflare and his mobile carrier, as he mentions in the quote above.
Jerry@feddit.online · 1 pts · 43d
No, the agent string is not enough.
I just checked. In the past 24 hours, the firewall has dumped 98K requests from AI bots that tried really hard to look like normal traffic and not like AI bots. These bots did a lot more than just set a user agent to try to look like real users, but it didn't help them.
Unattributed@feddit.online · 1 pts · 43d
Not surprising… I'm sure CloudFlare is working overtime to block these ridiculous bots. I've seen my analytics provider is blocking tons of bots every day, even low volume sites like mine are getting hit with hundreds of bot requests.