I’m using university journal portal OAI endpoints for the third time in a row to fix a typo and handle DOI lookups.
Does anyone know how disrespectful or intrusive this is considered to be?
A few years ago, I scraped arXiv and they blocked my IP almost immediately; perhaps if I had used arXiv’s OAI interface, they wouldn't have blocked me.
But I really have no idea just how disrespectful or intrusive this is!
2 Comments
xianjam@programming.dev · 2 pts · 18d
Basic etiquette, you'd probably want to ask the university portal maintainer.
Scraping is generally considered bad manners, and yes you would be immediately blocked if you ignore robots.txt. I assume by OAI you mean OpenAI? I'm unsure what kind of integration your university has, but like with many AI integrations, there is little oversight, which leads to... oversights.
obbeel@kbin.earth · 1 pts · 18d
OAI are API endpoints for metadata access of articles. I imagine API access is more respectful in terms of bandwidth since it's organized specifically for access by the universities?