[Solved] Cloudflare and (friendly!) bots

I'm writing a bot. I just created a Backrooms community (!backrooms@lemmy.world) and I want a bot to post links (as the user @backroomsbot@lemmy.world) to Backrooms-related content from the Backrooms wikis on a regular basis. (My current thought is daily, but I could be persuaded otherwise.) I might also add other features in the future like checking Kane Pixels' YouTube channel and automatically posting links to his new videos, but that's probably a later (if ever) kind of feature. I don't have any specific plans to add any kind of automod or mod tools capabilities at this point. (In fact, I don't have plans to make the bot account a mod in the community.) But I guess there might be a future when that happens.

I've read the Bot Usage and Guidelines section of the TOS and I've read this thread. It's all straightforward and I'm following all the rules and guidelines. My plan is not to put it on a cron for a while. I'll just run it manually every day for a while (until it proves itself) so that I can inspect what it does and if it misbehaves (not that I expect it to, but out of an abundance of caution) fix issues quickly to not impact users on Lemmy. I have also tested it on a locally-running Lemmy instance to ferret out any bugs or bad behaviors. It doesn't accept any commands from other users (well, except command-line arguments from me when I run it on my computer from a Bash prompt, but it doesn't take any actions in response to being @'d or DMed or anything). It DMs my user account when it encounters an error. I've been a professional software engineer for decades, so I like to think I know a little bit about what I'm doing. (Not that that means I'm incapable of producing bugs, but yeah.) And I cannot express in strong enough terms how cold and dead I will be in the ground before I put even a single solitary god-damned line of GenAI-generated code in any coding project I write. This is not some vibe-coded monstrosity. I wrought every single line using my brain, Go documentation, and Vim as God intended.

Oh, and I'll publish my bot's code on Codeberg or something under AGPL one of these days once it has demonstrated real-world success at doing what it does for a while.

All that to justify my use of "friendly" in the title of this post. Heh.

Anyway! To my question. My bot is being blocked by Cloudflare's CAPTCHA verification. You know, the thing specifically intended to repel bot traffic. Bots do seem to be explicitly allowed by .world's rules, so long as they follow all the rules. And I've definitely seen other bot accounts successfully posting. So my guess is that there's some good way to get my bot posting properly without being intercepted by Cloudflare. All that said, my searching hasn't revealed anything specific about how exactly that could be made to work. I'm hoping someone here can give me a lead or two, or just tell me why I'm wrong-headed about some basic assumption I'm making.

Thanks in advance!

Update: I got it working! Mostly the problem was that I wasn't caching my JWT token. Thanks again to MrKaplan.

5 points · 6 comments · view on lemmy.world

6 Comments

MrKaplan@lemmy.world · 4 pts · 9d (6 replies)

Hello,

use the official Lemmy API for automated interactions, don't use headless browsers or pretend to be a browser.

TootSweet@lemmy.world · 1 pts · 9d (5 replies)

I'm not using any headless browser or anything. I'm using Go's "net/http" package from the standard library which just... makes HTTP requests. I'm making requests to endpoints like https://lemmy.world/api/v3/user/login and https://lemmy.world/api/v3/post. The basic happy-path process is that every time it runs, it:

  • Gets the community id by name.
  • Logs in (getting the JWT token).
  • And then posts to the community in question (using the JWT token).

The user agent it uses is Go-http-client/1.1 if that helps any.

MrKaplan@lemmy.world · 3 pts · 9d (4 replies)

Then I don't know where you're seeing Cloudflare CAPTCHAs. There shouldn't be any on any of the API paths.

I've checked both URLs you mentioned above and only saw a Python user agent getting blocked with a browser integrity check on one of them in our CF logs. I've now added a dedicated rule to skip browser integrity checks on all API paths.

daychilde@lemmy.world · 1 pts · 8d
[ removed ]
TootSweet@lemmy.world · 1 pts · 8d (2 replies)

Hrm, well, now I'm noticing that I'm getting inconsistent results.

I keep getting {"error":"rate_limit_error"} from the login endpoint, and I've made... maybe 10 requests to that endpoint over the course of the day? And I've had that block this user (like, my @TootSweet account) from being able to log in. But I really don't think I'm sending any noticeable amount more traffic than I would be without the bot testing.

But sometimes I get past login and get a Cloudflare page back on another call. (In the unhappy path it tries to DM me, and it's tried that and failed.)

Maybe I'm just not waiting long enough between requests? I had it set to wait 3 seconds between requests, but maybe I'll bump it up to 5.

Thanks very much for your help so far. Sorry to throw more at you here. It seems the previous Cloudflare issue isn't what I'm running into any more at least.

MrKaplan@lemmy.world · 1 pts · 8d (1 reply)

You should persist the auth token, rate limits are by IP. They also don't exclude successful logins currently. This is a limitation in Lemmy. We have currently configured it to allow 5 logins per IP per day, but if you're lucky you can exceed that, as the consumption is not synchronized between backends at this time.

TootSweet@lemmy.world · 1 pts · 8d

Ok. That helps a lot. Thanks much! I'll take that back to my drawing board and I'd imagine that'll fix things for me.