Ireland’s government website, gov.ie, has a quite comprehensive robots.txt file, which by excludes almost all crawlers associated with AI training from scraping the site. One missing however is Mistral.
Ireland’s government website, gov.ie, has a quite comprehensive robots.txt file, which by excludes almost all crawlers associated with AI training from scraping the site. One missing however is Mistral.
2 Comments
breadsmasher@lemmy.world · 22 pts · 20d
jokes on them, LLM companies dont care about robots.txt
zod000@lemmy.dbzer0.com · 2 pts · 19d
Exactly, but I almost see this the other way. Not mentioning Mistral could just signal that they weren't important enough to even get mentioned.