@GroupNebula563@asklemmy@askfedi the one idea i had was a browser extension where users seeing an index-worthy page could click a button to manually add the DOM from their session to a curated index, after passing whatever anti scrape countermeasures were in use.
No crawling, just users voting "i think other people should be able to find this page".
The level of anti-bot defenses deployed across the internet in response to LLM crawlers is so high that it's basically impossible to create an independent search index in 2026. If you're not already one of a handful of trusted crawlers you're just gonna get cloudflare and anubis block pages.
@GroupNebula563 @asklemmy @askfedi the one idea i had was a browser extension where users seeing an index-worthy page could click a button to manually add the DOM from their session to a curated index, after passing whatever anti scrape countermeasures were in use.
No crawling, just users voting "i think other people should be able to find this page".
@GroupNebula563 @asklemmy @askfedi i am not aware of any, nor do i think it's likely that one exists.
The level of anti-bot defenses deployed across the internet in response to LLM crawlers is so high that it's basically impossible to create an independent search index in 2026. If you're not already one of a handful of trusted crawlers you're just gonna get cloudflare and anubis block pages.
@mattblaze@federate.social Would this qualify as... An antique antique shop?