Hi all, I selfhost private instance of Lemmy for my friends behind Pangolin reverse proxy. I noticed something interesting in the logs; Lemmy specifically gets pinged / tried to access each midnight UTC from what looks like an IP from inside the network. Just out of curiosity, do you have any idea what that could be? I have federation off and private instance on, but maybe it is something from Lemmy network checking if my server is alive? Thank you in advance
Update: So it turns out I was perhaps correct with my hunch. The local IP turns out to be the proxy I set for ports 80 and 443 (it was internal Wireguard IP). Unfortunately my current setup did not allow me to catch which IP the request came from (which is a problem I have to solve later) but the lemmy-proxy container got requests for GET /.well-known/nodeinfo and GET /nodeinfo/2.1. So it is probably something checking my server, likely from the Lemmy network.
Update 2: So after I disabled Pangolin for one night, after I reenabled it, the requests do not come again! So the Lemmy network must have figured out that my instance is set to private and stopped pinging.
17 Comments
renegadespork@lemmy.jelliefrontier.net · 31 pts · 323d
How could we tell you about an IP inside your own network? Look at the host using that IP and see what’s running on it.
removerpuzzlehunchback@lemmy.world · 2 pts · 323d
Well it is definitely specific to Lemmy, I selfhost over 20 services and only Lemmy gets pinged on midnight. The only other service I saw doing this was Nextcloud, Nextcloud instance needs to reach itself, but for Lemmy it is a different IP, which is puzzling me
mesamunefire@piefed.social · 9 pts · 323d
Got the log?
removerpuzzlehunchback@lemmy.world · 2 pts · 323d
Nothing in Lemmy's logs, in Pangolin's logs it's only the lines about attempted access each midnight
ClickyMcTicker@hachyderm.io · 5 pts · 323d
@removerpuzzlehunchback Logs? Details? Anything? Your wild uninformed guess is as good as my wild uninformed guess.
removerpuzzlehunchback@lemmy.world · 1 pts · 323d
Sorry that I have not provided more details, I thought that this behaviour is specific to Lemmy and I was just curious what causes it. There isn't anything interesting in the logs as the traffic gets stopped at Pangolin's level, so it's only a logline in Pangolin's log and nothing in Lemmy's. I guess I could turn off Pangolin auth for one midnight and see in Lemmy's log if I catch something.
slazer2au@lemmy.world · 5 pts · 323d
An ICMP ping or a web request?
If it's a web request the first thing that comes to mind is do you have BitWarden?
removerpuzzlehunchback@lemmy.world · 3 pts · 323d
Yes, I do. It is probably a web request
slazer2au@lemmy.world · 4 pts · 323d
There was a post a few days ago about someone using it and it pulled a tonne of data. I wonder if it also does polls to check if the link is still valid.
Appoxo@lemmy.dbzer0.com · 7 pts · 323d
Bitwarden uses the favicon from the first link in the password entry.
For my selfhosted web pages I use the public info page of the selfhosted page (e.g. openMediaVault) and set detection to [
none].This way it won't match against the 3rd party page but I get the icon :)
BUUUT it should only poll if you activate the program/extension.
Don't know why it should poll at midnight
removerpuzzlehunchback@lemmy.world · 3 pts · 323d
I do not have my Lemmy's link in the Bitwarden
Cyber@feddit.uk · 3 pts · 323d
And when you ping that IP address back, what happens?
Can you trace it?
Maybe setup wireshark and record what happens at that time of night...
removerpuzzlehunchback@lemmy.world · 1 pts · 323d
I will definitely do that, right now I can't work with anything because the traffic gets stopped at Pangolin's level, but I will turn off Pangolin's auth for one night
EarMaster@lemmy.world · 3 pts · 323d
Is your server running on UTC? Depending on your location midnight UTC could also be 8 AM and it could be a user with a very regular morning schedule.
Only you can find out which machine is sending this request...
removerpuzzlehunchback@lemmy.world · 1 pts · 323d
My timezone is CET, so I get the ping on 2AM. The Lemmy container should be on UTC as I did not specify the timezone when launching the container. It is definitely not human, as the ping comes exactly on midnight UTC, or seconds away from midnight. I will turn off the Pangolin auth and investigate further this midnight. Again sorry for not providing more information, I was certain that it is a thing internal to Lemmy and I was just curious what it is
removerpuzzlehunchback@lemmy.world · 2 pts · 322d
Update: So it turns out I was perhaps correct with my hunch. The local IP turns out to be the proxy I set for ports 80 and 443 (it was internal Wireguard IP). Unfortunately my current setup did not allow me to catch which IP the request came from (which is a problem I have to solve later) but the lemmy-proxy container got requests for GET /.well-known/nodeinfo and GET /nodeinfo/2.1. So it is probably something checking my server, likely from the Lemmy network.
Jayjader@jlai.lu · 2 pts · 321d
Sounds like either federation working as intended, or some client app trying to cache info about your instance. Might be https://fedidb.com/ or https://fediverse.observer/ or some other service.