How to Build a Powerful Reverse Proxy Firewall for Blocking the Evil Web-Scraping Robot Hordes from Hell

https://cheapskatesguide.org/articles/debian-netinstall-waf.html

My goals for this firewall were mostly to provide better robot blocking and perhaps some more powerful DDoS protection than my Raspberry Pi 3 web server is capable of delivering. I still have to do some testing before I will know if my new firewall actually provides either of those, but at least I now have the additional ability to run multiple physical web servers on my LAN. Exploring that should be fun, and fun is a very important component of running a home web server.

Not my article. Just sharing.

305 points · 55 comments · view on lemmy.world

55 Comments

Auth@lemmy.world · 95 pts · 351d (14 replies)

All this effort to block clankers? Absolutely worth it fuck clankers.

henfredemars@infosec.pub · 53 pts · 351d (13 replies)

I absolutely love the term clankers. It's the perfect blend of dystopian cyberpunk and the very real threat of AI.

Vupware@lemmy.zip · 11 pts · 351d (12 replies)

It seems goofy to me — I wish we had collectively picked a term with more oomph.

I’m struggling to come up with an alternative though.

HeyThisIsntTheYMCA@lemmy.world · 9 pts · 350d (4 replies)

toasters

massacre@lemmy.world · 3 pts · 350d (3 replies)

So say we all.

HeyThisIsntTheYMCA@lemmy.world · 4 pts · 350d (2 replies)

so say we all

No_Eponym@lemmy.ca · 2 pts · 350d (1 reply)

All of this has happened before.

HeyThisIsntTheYMCA@lemmy.world · 1 pts · 350d

does that mean we get to burn down textile mills?

addie@feddit.uk · 4 pts · 351d (3 replies)

I'm still trying to make 'sloppers' happen. Perfectly describes the lack of thought that goes into what they produce.

4grams@awful.systems · 11 pts · 350d

I like 'sloppers' as a term for the morons distrubing and consuming the shit that the clankers are excreting.

ICastFist@programming.dev · 3 pts · 350d

clankers make the end result that sloppers (meatbags) eat up ;)

anomnom@sh.itjust.works · 3 pts · 350d

Yeah and the make a mess of network traffic that slows everything down.

Sludgers works too, but I like slop for the LLM output, so it makes sense as the bot term of derision.

mhague@lemmy.world · 0 pts · 350d (2 replies)

Stealing a slur from Star Wars and engaging in traditional name calling to show we disapprove of uncreative slop.

We can't even think of an original term. We can't think of a novel way to shit on AI. We just copy what everyone else is doing to make fun of the plagiarism machine.

Shdwdrgn@mander.xyz · 7 pts · 350d

Why does everyone think the term came from Star Wars? I know it was used in steampunk before then, and google suggests it goes back to a 1958 article about robots. Sorry, not trying to be pedantic, just feels like a lot of people give Star Wars unjust credit for things they didn't actually create.

gravitas_deficiency@sh.itjust.works · 6 pts · 350d

No no - it’s not plagiarism; it’s standardization.

BaroqueInMind@piefed.social · 33 pts · 351d (14 replies)

This page isn't loading for me.

Zachariah@lemmy.world · 82 pts · 351d (13 replies)

found the bot

BaroqueInMind@piefed.social · 21 pts · 351d (12 replies)

I'm getting a 404 error, using Cloudflare DNS, who ironically has the best commercial clanker protection in the world, otherwise half the world's internet wouldn't use them

possiblylinux127@lemmy.zip · 17 pts · 351d (8 replies)

Bot

BaroqueInMind@piefed.social · 1 pts · 350d (7 replies)

Not a bot. Both of you can go fuck yourselves with an ENTIRE can of bear mace.

possiblylinux127@lemmy.zip · 6 pts · 350d

Sounds like something a bot would say

AtariDump@lemmy.world · 5 pts · 350d (5 replies)

Using a VPN that you forgot is on?

BaroqueInMind@piefed.social · 0 pts · 350d (4 replies)

Yep. Why do you ask?

AtariDump@lemmy.world · 5 pts · 350d

Because the VPN might be the reason you’re being blocked from the page.

Jason2357@lemmy.ca · -2 pts · 350d (2 replies)

Cloudflare is a protection racket. They cover so many websites because it’s easier to pay the mafia.

BaroqueInMind@piefed.social · 8 pts · 350d (1 reply)

Im not a cloudflare dick rider, so if you have a suggestion for a better service with commensurate features, im all ears.

sirboozebum@lemmy.world · 1 pts · 345d

I thought there was a self-hosted service that was an alternative to cloudflare tunnels?

jaybone@lemmy.zip · 32 pts · 351d (9 replies)

Should we just move everything to tor and start the Internet over again?

HeyThisIsntTheYMCA@lemmy.world · 10 pts · 350d (1 reply)

look if i can't browse from my fridge i don't want to know about it

zqps@sh.itjust.works · 2 pts · 350d

Not sure about Tor but your fridge definitely supports mesh networks whether it wants to or not.

maxwells_daemon@lemmy.world · 7 pts · 351d (5 replies)

Wait till you hear about betanet

kata1yst@sh.itjust.works · 12 pts · 350d

I was interested until I saw the crypto stakes to vote on changes baked in.

BuboScandiacus@mander.xyz · 8 pts · 351d (3 replies)

Isn't that the new freenet ?

0_o7@lemmy.dbzer0.com · 8 pts · 350d (2 replies)

No that's alphanet

cryptTurtle@piefed.social · 6 pts · 350d

surely you mean Pipernet

BuboScandiacus@mander.xyz · 2 pts · 350d

Damn it, with their confusing naming !

CephalonKappa@discuss.tchncs.de · 4 pts · 350d

*i2p

quick_snail@feddit.nl · 23 pts · 351d

403 error. Congrats, you built a broken system of false-positives.

GraveyardOrbit@lemmy.zip · 15 pts · 351d (2 replies)
[ removed ]
BaroqueInMind@piefed.social · 12 pts · 350d (1 reply)

Opnsense > pfsense

The fact that I have to go through a fucking purchase page, even though pfsense is free (for now), is sketch as hell. First step in their inevitable enshittification.

Opnsense is funded by European non-profits, and is has a better UI

ms_lane@lemmy.world · 3 pts · 350d

Not to mention they tried running a disinformation campaign against OPNsense for a few years, which was resolved in court.

Also they implemented a WireGuard module that after a review upstream on FreeBSD was found to be completely hocus.

gravitywell@sh.itjust.works · 12 pts · 351d (9 replies)

Well it cant be that good becauase it thinks im a bot.

Anubis works pretty well for me so far in blocking clankers.

m33@lemmy.zip · 9 pts · 351d (6 replies)

It seems Anubis’github issues shows many false positives with smartphone browsers. Depending on OP’s target audience it’s worth to hunt for FP

gravitywell@sh.itjust.works · 4 pts · 351d (5 replies)

I just wish i could read it, it seems to block based on my IP which isn't really a good way to identify bots.

m33@lemmy.zip · 3 pts · 351d (3 replies)

Here is a mirror https://git.qiuwen.net.cn/Mirror/anubis handle with care

gravitywell@sh.itjust.works · 7 pts · 351d (2 replies)

Thanks but i meant the site in the original post https://cheapskatesguide.org/articles/debian-netinstall-waf.html

It says

403 Error Your IP address has been blocked. This MAY be because you have made yourself look like a robot by using an unknown VPN or Tor exit node.

rumba@lemmy.zip · 7 pts · 351d

Blocking tor is pretty bold, that network is too slow to use for anything but straight up privacy.

m33@lemmy.zip · 2 pts · 351d

How ironic

I guess you will have to resort to online translators or actual web proxies to read these pages 🙄

SteveTech@programming.dev · 2 pts · 350d
0_o7@lemmy.dbzer0.com · 3 pts · 350d (1 reply)
gravitywell@sh.itjust.works · 2 pts · 350d

Oh interesting! Thank you.

m33@lemmy.zip · 8 pts · 351d

And then you have to fill a block list with something like https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker

quick_snail@feddit.nl · -4 pts · 351d

Aka "how to harm marginalized folks" and prevent them from accessing your content too