c/technology · by aary@xn--e1aghfa.xn--e1aqbccjfc.xn--p1acf · 94dI want to find users of Heritrix3 engine I have such setup, but with some problems: it always does full crawl (doesn't do deduplication) I am unable to control crawl order (don't have enough knowledge) 3 points · 1 comments · view on lemmy.world
1 Comments
Вежливый_Лис@xn--e1aghfa.xn--e1aqbccjfc.xn--p1acf · 2 pts · 94d
kreuzcrawl — a high-performance Rust web crawling engine.