Crumb
c/technology · by BrikoX@lemmy.zip · 1d

99% of My Website Traffic Is Bots | PatronView

https://patronview.com/news/99-percent-of-my-website-traffic-is-bots/

A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what's working now.

64 points · 14 comments · view on lemmy.world

14 Comments

BlameTheAntifa@lemmy.world · 19 pts · 1d (3 replies)

I tried to give them a human view, but it appears that they block VPNs so I gave up.

snoons@lemmy.ca · 26 pts · 1d (2 replies)

It's a vibe-coder ai bro dude guy complaining about ai.

but also I have IVPN and it loaded without issue.

kazerniel@piefed.zip · 15 pts · 1d

It’s a vibe-coder ai bro dude guy complaining about ai.

the irony is rich...

bugbear@lemmy.sdf.org · 2 pts · 7h

thanks, you saved me time opening that link

codexarcanum@lemmy.dbzer0.com · 11 pts · 1d

I work on an ecomm site and it's nightmarish. We have all the cloudflare protections turned on but do try to tune out system so american AI companies will scrape us (we want chatgpt to send us customers and use our data).

Still, we have a country allow list because it's easier to admit North American traffic and the specific IPs our contractors use, then block everyone else. The amount of bot traffic flooding in globally is insane. I don't think the current day internet infrastructure is going to be able to absorb the abuse much longer.

fubarx@lemmy.world · 5 pts · 1d

I thought Cloudflare's anti-AI defenses were meant to dynamically catch all of this. That's a lot of work to do to keep bot traffic away.

FWIW, I tried loading it with a VPN, it showed the content, but not all the images. Turned off VPN, it fully loaded. Maybe the blog has different rules.

elvith@feddit.org · 2 pts · 1d

Wait.... AJ12Bot and Ahrefsbot are still around?! I have them blacklisted since... At least 2012 now, I think?

I usually don't really evaluate which bots might or might not be around anymore and still keep the rules for them, LOL

melroy@kbin.melroy.org · 2 pts · 1d (6 replies)

I feel you! Same issue here. I created my own solution https://angieguardian.org/. Fully open source and free to use under agpl license.

undefinedTruth@lemmy.zip · 3 pts · 12h (1 reply)

Why would a website owner use this over Anubis. Not bashing it or anything, just curious how it differentiates?

melroy@kbin.melroy.org · 2 pts · 11h

I wanted to have high performance. See my benchmarks. I'm able to reach 150k-180k reqs/sec allowing me to handle ddos attacks as well. Even under very heavy load. Anubis will fall apart.

Great question BTW!

onlinepersona@programming.dev · 1 pts · 1d (1 reply)

Is that like the anime girl proof we see everywhere?

melroy@kbin.melroy.org · 5 pts · 1d

No that's Anubis

BrikoX@lemmy.zip · 0 pts · 1d (1 reply)

I don't run PatronView, just sharing the blog post from them.

I tried to access angieguardian.org, but was met with "JavaScript is required to access this site" which basically guaranteed I will never visit it again.

melroy@kbin.melroy.org · 2 pts · 1d

OK fine. That is due to the pow. The whole point of Angie Guardian. I also don't like it. But the internet is fked up!