Crumb
c/news · by ccunning@lemmy.world · 350d

Pay-per-output? AI firms blindsided by beefed up robots.txt instructions.

https://arstechnica.com/tech-policy/2025/09/pay-per-output-ai-firms-blindsided-by-beefed-up-robots-txt-instructions/?utm_social-type=owned

25 points · 8 comments · view on lemmy.world

8 Comments

whiwake@lemmy.cafe · 7 pts · 350d (6 replies)

lol. AI firms stop reading robots.txt

ccunning@lemmy.world · 5 pts · 350d (5 replies)

I think the idea is that all parties would find it beneficial so they would be incentivized to read it:

Leeds told Ars that the RSL standard doesn't just benefit publishers, though. It also solves a problem for AI companies, which have complained in litigation over AI scraping that there is no effective way to license content across the web.

whiwake@lemmy.cafe · 5 pts · 350d (4 replies)

I imagine that when it comes to scraping data, the benefits of having the data far out away the risk of getting sued for scraping it.

SpookyMulder@twun.io · 5 pts · 350d (1 reply)

Not to be That Guy, but with kindness I offer a small correction: "out away" -> "outweigh".

whiwake@lemmy.cafe · 1 pts · 350d

Thank you, I was talking to the phone and it did not know what out we were weighing.

ccunning@lemmy.world · 3 pts · 350d (1 reply)

If nothing else it gets rid of one of the arguments they’re currently using in their defense at trial.

whiwake@lemmy.cafe · 2 pts · 350d

I hear you, and totally agree. I just remain a bit skeptical that something like this will have any effect as long as AI is making money hand over fist

TheMightyCat@ani.social · 2 pts · 350d

If there is a way to combine this with something like anubis this could be very interesting, then you are also not dependent on the honesty of reading the robots.txt

Have a RSL license? here you go scrape the content.

Trying to freeload? anubis time.