cross-posted from: https://lemmy.sdf.org/post/58474758
Someone brought the HTtrack app to my attention in this thread. Superficially it’s a great concept. But it did not work for me.
The author of HTtrack says they respect the
robots.txtfiles. I’m not sure if this is my problem. But what an absurd line to draw. Why should a realtime GUI user have more privilege to view a website than a CLI user?If anything, it should be the other way around.
- People who lack the priviledge of having Internet at home face this nasty discrimination of being treated like a bot after they make the extra effort of commuting to a public library just to fetch a website for offline viewing later. This 2nd-classing of a demographic who is already marginalized is quite despicable.
- People with Internet at home can schedule HTtrack to run at an off-peak time with a narrow bandwidth put less burden on the server than the realtime GUI users who obviously hit the site mostly during peak times.
- (update) Some people are on measured rate Internet connections that give tiny daytime quotas and generous late night quotas. Tools like HTtrack are needed to manage this. Which ultimately benefits the more privileged Internet users who have no constraints.
Update
There is a poorly worded -s0 option to ignore robots.txt. Fooled some people into thinking the tool uses robots.txt to direct the fetches.
14 Comments
grue@lemmy.world · 11 pts · 3d
robots.txthas always been nothing more than a suggestion. It has fuck-all to do with GUI vs CLI; it's only relevant to HTtrack because the developer apparently chooses to be polite and follow it.Don't like it? Fork it and make an impolite version.
evenwicht@lemmy.sdf.org · -7 pts · 3d
Indeed I know it is voluntary. I’m calling out a shit decision by the developer who produced a tool that is unlikely to be used as a data harvesting AI op. The author knew his tool in practice would generally be no more robotic than Firefox itself. It’s not scraping. It’s fetching. The tool simply separates the fetching and viewing activities.
Of course it does. GUI users aren’t getting Firefox versions that yield to robots.txt files, but CLI (httrack) users are.
Of course, but that’s not the point.
(edit) there is a confusing
-s0option to ignore robots.txt.historicaldocuments@lemmy.world · 5 pts · 3d
A lot of the world runs on gentlemen's agreements and handshake deals. There's no real way to have a robots.txt for a public facing internet without starting an arms race with the server having rate limiting, metrics to check and see if someone's going alphabetically (for example), captchas on every page to verify it's a human, which the scraper will counter with a randomized scrape list, LLMs to pass captchas, etc. The server could institute some sort of login, but that defeats the purpose of an open internet. So robots.txt can't be enforced from a technological standpoint. One could glean information from who's ignoring it and obviously scraping, but that's about it.
There's no technological difference between these if the scraper wants. I was going to deep dive htrack because I've never heard of it before, but stopped when I saw some of it's options. It's straight up a website scraper. It not only has options to ignore robots.txt, but it also has a setting that apparently uses robots.txt to know where to scrape! Wrap a systemd timer around it and start working on your own archive.org.
calliope@retrolemmy.com · 5 pts · 3d
Agreed with everything you said, I just wanted to modify this
It’s important to note that there’s no difference at all from the server’s side, other than an implication of how often scraping happens or how the data is being used.
Which is why the GUI restriction was added to robots.txt in the first place. From the server’s perspective, most command-line applications have historically been some kind of “robot” instead of something more interesting from the server owner’s perspective.
This is hilarious.
historicaldocuments@lemmy.world · 2 pts · 2d
I thought about maybe checking server logs for what files a scraper might skip just for bandwidth or whatever, but I don't know much about that world.
I had no idea there was a GUI restriction since I stopped digging pretty quickly about it. Upon further reading it's apparently been used in several lawsuits successfully as a sort of "no tresspassing" sign with some court precedent to it. Neat.
evenwicht@lemmy.sdf.org · 1 pts · 3d
^ THIS
evenwicht@lemmy.sdf.org · 1 pts · 3d
Bingo.
Not on to advocate for internet open just for people privileged to have access in their homes.
Are you calling Firefox a scraper then?
Consider there is:
Why would you call “2” a scraper and “1” otherwise?
Nonsense. The author was sloppy in their phrasing. That is not what he means by “follow robots.txt”. By “follow”, he means “obey”, not visit. I was also initially fooled by the same ambiguous wording when I first saw it.
ICastFist@programming.dev · 4 pts · 3d
Real question, what exactly are you doing there? Why not access the site with a browser, CTRL+S, and move to the next page? You're getting an internet connection off the library, so it's either a wifi or through a computer of theirs.
evenwicht@lemmy.sdf.org · -1 pts · 3d
I enter the library with a laptop, and list of tasks and URLs. Tasks, meaning I have to e.g. search for a PDF manual for a 2nd-hand appliance I either bought for pulled from a dumpster. Or research something. For URLs that I just need to save for later reading, I open them in FF (many tabs) and use the SingleFile extension to save them one by one. Of course that robs me of human time that I need for tasks that must be interactive. My time would be more wisely managed to have HTtrack fetching what I need in the background while I do interactive things. I sometimes stay until I get kicked out because the library is closing, in which case my needs were not all satisfied.
For Lemmy, I save posts in advance as text files and copy-paste the text into a Lemmy web client. This is also not a good use of my time but the only offline lemmy client is broken. But if there were a non-broken lemmy client for offline access, it would probably face the same discrimination by this reckless and obnoxious anti-bot movement.
historicaldocuments@lemmy.world · 1 pts · 2d
Gotcha.
I'm sure there's a way to wire it up that way. It's bound to have a Turing complete language in it somewhere and by definition it has all the tools to copy down files from the internet. If I were going into the scraping business I'd personally start with a scraper and give it a Firefox user agent. In short, scraping is about intent as much as tooling, and there's no way to convey and enforce intent over HTTP. From a practical standpoint there's a spectrum of activities that approximately go from Browsing to Scraping to DDoS in that order that really only differ in terms of scale.
If I'm hosting content then if #2 is a scraper and #1 is a browser, and they don't cost me anything different then I don't really care (someone else who's done that kind of stuff can weigh in if I'm wrong). What a lot of sites seem to be reporting is that they have a small, steady hosting cost per month until WHOOM! an AI company scrapes their whole site and their hosting cost jumps that month, then the next month and the next as well because companies that would destroy copies of rare books to feed to their training don't care. Facebook got called out sort of recently for scraping some entire pornsite's content. It sounds like what you're describing is just timeshifted browsing. I doubt anyone notices unless you're really describing what you're trying to do incorrectly.
You'll often hear/read "follow" used by itself to imply "follow the rules," in this case "follow the rules specified by robots.txt."
From a technical standpoint an LLM recommends trying this code snippet to make httrack ignore the robots.txt file:
I haven't tried it personally.
evenwicht@lemmy.sdf.org · 1 pts · 2d
Yes, you simply turn off images. Then Firefox is a scraper from the server standpoint. I use FF that way when operating under a bandwidth quota. Scraping a page is far less of a burden on the server than fetching the whole page (images, js, and all) for GUI rendering.
Turing completeness is wholly irrelevant. A fetch operation does not absolutely need a turing complete language. But you will struggle to render the content for a graphical presentation without a turing complete language.
Not “in that order”. Browsing graphically causes every image to be fetched. It’s much heavier than the HTML text that a scraper pulls. 10,000 gui users each fetching a page will strain your webserver faster than 10,000 html text pages being scraped.
Non-graphical users suffer from marginalization driven by the shitty attitude that “bots are bad and humans should alter their behavior and fetch images to look less like a bot as we don’t give a shit about oppressing bots.”
Timeshifted browsing is purpose of HTtrack. An AI operation doesn’t use HTtrack which is designed to organize the browsing session into interactive projects which launch a local GUI browser for offline viewing. An AI op has zero interest in that. A scraping operation uses curl, wget, beautiful soup/python, etc.
It’s this kind of commentary that distorts public perception.
Scraper bots themselves serve many users, often more than the original site could afford to host. A scraper bot that grabs just raw text builds a database and makes the information more useful to more people than the original site. E.g. instead of each traveler manually visiting the website of every single airline (a costly image-rich bloat high in CO₂ emissions), they can go to a site that has scraped Ryanair and others. The aggregated text info is more useful. Consumers then only have to visit /one/ airline website after seeing all the info side-by-side. This is much less strain on resources.
That’s what I was saying. Hence why the claim that HTtrack follows locations listed in
robots.txtis nonsense.I appreciate the tip. Indeed
-s0is what I worked out that I needed. You seem bent against AI so it’s a bit unexpected that you would be using it. I avoid AI because I generally boycott the corporations pushing the LLMs. But I’m not generally opposed to LLM tech. I have a local language translator and probably need to look into other LLMs that can run locally.historicaldocuments@lemmy.world · 1 pts · 2d
I'll stop then. Have a nice day.
ICastFist@programming.dev · 5 pts · 3d
The vast majority of bots run headless, aka without a GUI. Actual people that use CLI to connect to a website are probably like 1% of the CLI connection cases
evenwicht@lemmy.sdf.org · 0 pts · 3d
Indeed this is how marginalization works. The smaller the demographic, the easier it is to discriminate against them and exclude them from society. It’s an oppressive mentality.