Search engines work by building an index over all the sites on the internet. The index and the status of the sites aren't linked, the search engine basically has a cached view of the site.
If the site goes down after it is indexed, DDG won't know about it until the next index.
Stale index: the link and summary DDG has stored no longer matches the site's content because the site changed it. The index will get updated eventually. This is the most likely explanation.
Selective content: the site dynamically chose to serve you different content than it serves to DDG's crawler or other visitors. It might, for example think that you're a bot and want you to go away.
404 means that a web sever doesn't have something at the URL you requested.
Search engines build an index periodically by looking at what's on a website. That's what provides your search results. They aren't informed if a web server removes or moves content, though eventually, they'll have a bot that checks, notices the change, and updates the index.
They --- well, Google, and I assume most --- will update major sites more frequently, so this is probably more common if you're pulling something off a small, backwater website.
At first I thought “how is this relevant?” but increasingly yeah, us web developers to use HTTP 404 rather than HTTP 403 increasingly as it can be more secure to pretend something doesn’t exist than to explicitly say “you don’t have access to this right that exists.”
I’ve never especially loved the practice but I’ve done it at work for really sensitive endpoints.
14 Comments
CameronDev@programming.dev · 16 pts · 19h
Search engines work by building an index over all the sites on the internet. The index and the status of the sites aren't linked, the search engine basically has a cached view of the site.
If the site goes down after it is indexed, DDG won't know about it until the next index.
village604@adultswim.fan · 4 pts · 19h
DDG uses Bing on the back end for the most part. They don't maintain an index.
CameronDev@programming.dev · 2 pts · 18h
DDG is just using Bings index, which means DDG still wont know about any changes until Bing reindexes.
one_old_coder@piefed.social · -5 pts · 18h
Do you work for them?
village604@adultswim.fan · 5 pts · 17h
I'm not sure why my comment would make you think I'm an employee.
one_old_coder@piefed.social · 0 pts · 17h
How can you claim that "DDG is just using Bings index" then?
randombullet@programming.dev · 9 pts · 16h
https://duckduckgo.com/duckduckgo-help-pages/results/sources
Because they can look things up?
schipelblorp@sh.itjust.works · 2 pts · 9h
Are those internal documents?? /s
Zak@lemmy.world · 8 pts · 18h
remon@ani.social · 6 pts · 19h
If even you know where someone lives, sometimes they won't be home when you're trying to visit.
tal@lemmy.today · 3 pts · 13h
404 means that a web sever doesn't have something at the URL you requested.
Search engines build an index periodically by looking at what's on a website. That's what provides your search results. They aren't informed if a web server removes or moves content, though eventually, they'll have a bot that checks, notices the change, and updates the index.
They --- well, Google, and I assume most --- will update major sites more frequently, so this is probably more common if you're pulling something off a small, backwater website.
one_old_coder@piefed.social · 1 pts · 18h
Give us the link and query.
sic_semper_tyrannis@lemmy.today · 1 pts · 18h
Do you have a VPN on?
undefined@lemmy.hogru.ch · 1 pts · 2h
At first I thought “how is this relevant?” but increasingly yeah, us web developers to use HTTP 404 rather than HTTP 403 increasingly as it can be more secure to pretend something doesn’t exist than to explicitly say “you don’t have access to this right that exists.”
I’ve never especially loved the practice but I’ve done it at work for really sensitive endpoints.