Many consider AI bots as very bad because they ignore established guidelines when crawling a given site. There is a well known and respected way sites can tell good bots how to behave, called robots.txt. Any website can create one and expect good bots (Google, Bing, Facebook, etc) to honor it. For example https://lemmy.world/robots.txt tells bots what paths it should NOT crawl, and the maximum speed at which it should crawl the site.
Many AI bots have been seen completely disregarding this established standard.
It's a huge company, and I worked in a small subsection of it. For the most part, our team was fairly detatched from the main operations. It was good working there, the corporate BS didn't touch us too much and the people were great. I left for a job that was more Linux and networking focused. Not sure if that was a good move entirely, looking back. But if I stayed, I'd probably be getting bored if I didn't move into a different role
I’d consider search engine bots from the likes of Google, Microsoft, etc. to be benign. They provide a useful service to websites and honor robots.txt directives.
Bots that impersonate Google etc. and/or ignore robots.txt are considered bad bots.
10 Comments
ghodawalaaman@programming.dev · 10 pts · 102d
/r/data-is-not-beautiful :(
squirrel@cake.kobel.fyi · 6 pts · 102d
/c/data-is-not-beautiful
M33@piefed.world · 6 pts · 102d
Does AI scrapers and agents count as bae, or good bots ?
IphtashuFitz@lemmy.world · 3 pts · 101d
Many consider AI bots as very bad because they ignore established guidelines when crawling a given site. There is a well known and respected way sites can tell good bots how to behave, called robots.txt. Any website can create one and expect good bots (Google, Bing, Facebook, etc) to honor it. For example https://lemmy.world/robots.txt tells bots what paths it should NOT crawl, and the maximum speed at which it should crawl the site.
Many AI bots have been seen completely disregarding this established standard.
luthis@lemmy.nz · 3 pts · 102d
I used to work for Thales
terradragon@slrpnk.net · 2 pts · 101d
How was work there? Why'd you leave?
luthis@lemmy.nz · 4 pts · 101d
It's a huge company, and I worked in a small subsection of it. For the most part, our team was fairly detatched from the main operations. It was good working there, the corporate BS didn't touch us too much and the people were great. I left for a job that was more Linux and networking focused. Not sure if that was a good move entirely, looking back. But if I stayed, I'd probably be getting bored if I didn't move into a different role
Sixseven@sh.itjust.works · 2 pts · 101d
What are examples of benign bots?
Armok_the_bunny@lemmy.world · 1 pts · 101d
Google's search indexer.
IphtashuFitz@lemmy.world · 1 pts · 101d
I’d consider search engine bots from the likes of Google, Microsoft, etc. to be benign. They provide a useful service to websites and honor robots.txt directives.
Bots that impersonate Google etc. and/or ignore robots.txt are considered bad bots.