How do we know if there aren't a bunch of more undetected backdoors?

I have been thinking about self-hosting my personal photos on my linux server. After the recent backdoor was detected I'm more hesitant to do so especially because i'm no security expert and don't have the time and knowledge to audit my server. All I've done so far is disabling password logins and changing the ssh port. I'm wondering if there are more backdoors and if new ones are made I can't respond in time. Appreciate your thoughts on this for an ordinary user.

221 points · 79 comments · view on lemmy.world

79 Comments

mwguy@infosec.pub · 156 pts · 2y

"We don't" is the short answer. It's unfortunate, but true.

rotopenguin@infosec.pub · 60 pts · 2y (2 replies)

How do you know there isn't a logic bug that spills server secrets through an uninitialized buffer? How do you know there isn't an enterprise login token signing key that accidentally works for any account in-or-out of that enterprise (hard mode: logging costs more than your org makes all year)? How do you know that your processor doesn't leak information across security contexts? How do you know that your NAS appliance doesn't have a master login?

This was a really, really close one that was averted by two things. A total fucking nerd looked way too hard into a trivial performance problem, and saw something a bit hinky. And, just as importantly, the systemd devs had no idea that anything was going on, but somebody got an itchy feeling about the size of systemd's dependencies and decided to clean it up. This completely blew up the attacker's timetable. Jia Tan had to ship too fast, with code that wasn't quite bulletproof (5.6.0 is what was detected, 5.6.1 would have gotten away with it).

rotopenguin@infosec.pub · 28 pts · 2y (1 reply)

In the coming weeks, you will know if this attacker recycled any techniques in other attacks. People have furiously ripped this attack apart, and are on the hunt for anything else like it out there. If Jia has other naughty projects out here and didn't make them 100% from scratch, everything is going to get burned.

rotopenguin@infosec.pub · 2 pts · 2y

I think the best assurance is - even spies have to obey certain realities about what they do. Developing this backdoor costs money and manpower (but we don't care about the money, we can just print more lol). If you're a spy, you want to know somebody else's secrets. But what you really want, what makes those secrets really valuable, is if the other guy thinks that their secret is still a secret. You can use this tool too much, and at some point it's going to "break". It's going to get caught in the act, or somebody is going to connect enough dots to realize that their software is acting wrong, or some other spying-operational failure. Unlike any other piece of software, this espionage software wears out. If you keep on using it until it "breaks", you don't just lose the ability to steal future secrets. Anybody that you already stole secrets from gets to find out that "their secrets are no longer secret", too.

Anyways, I think that the "I know, and you don't know that I know" aspect of espionage is one of those things that makes spooks, even when they have a God Exploit, be very cautious about where they use it. So, this isn't the sort of thing that you're likely to see.

What you will see is the "commercial" world of cyberattacks, which is just an endless deluge of cryptolockers until the end of time.

MangoPenguin@lemmy.blahaj.zone · 53 pts · 2y (2 replies)

Self hosting personal photos doesn't generally require opening anything up to the internet, so most backdoors would not be accessible by anyone but you.

maxprime@lemmy.ml · 15 pts · 2y (1 reply)

Or someone who has penetrated your network.

JustEnoughDucks@feddit.nl · 4 pts · 2y

Of which the chances are slim to none for 99% of people simply because they aren't interesting enough to be a target beyond phishing, scans, and broad attacks.

Lmaydev@programming.dev · 43 pts · 2y (1 reply)

You don't. Hackers often exploit things like this for ages before they are found. Every bit of non simple software also has bugs.

But chances are you won't be the target.

Just keep everything updated and you should be alright.

possiblylinux127@lemmy.zip · 5 pts · 2y

And implement least privilege and automatic updates

VinesNFluff@pawb.social · 42 pts · 2y (2 replies)

Cheeky answer:

Actual answer:
Theoretically anyway, open source software's guarantee of "no backdoor" is that the code is auditable, and you could study it and know if it has any holes and where. Of course, that presumes that you have the knowledge AND time to actually go and study thousands of lines of code. Unrealistic.
Slightly less guaranteed but still good enough to calm my mind, is the idea that there is a whole-ass community of people who do know their shit and who are constantly checking this.

Do note that like. Closed source software is known to be backdoored, only, the backdoors are mostly meant for either the owners of the software (check the fine print folks) or worse, the governments.

The biggest thing that you should note is that: It is unlikely that you (or I or most of the people here) are interesting enough that anyone will actually exploit those vulnerabilities to personally fuck you over. Your photos aren't interesting enough except as part of a mass database (which is why Google/Facebook want them). Same for your personal work data and shit.

Unless those backdoors could be used to turn your machine into a zombie for some money-making scheme (crypto or whatever) OR you're connected to people in power OR you personally piss off someone who is a hacker -- it is very unlikely you'll get screwed over due to those vulnerabilities :P

bigmclargehuge@lemmy.world · 18 pts · 2y (1 reply)
[ removed ]
Tlaloc_Temporal@lemmy.ca · 2 pts · 2y

Even better: there are also backdoors in closed-source software that will never be found. There may be fewer backdoors inserted, but the ones that get in there are far more likely to stay undiscovered.

shortwavesurfer@monero.town · 35 pts · 2y

I would say you can't, but if you are using open source software, then somebody can and will find them eventually and they will be patched. Unlike with closed source software, you will never know if it has a backdoor or not. This whole episode shows both the problems with open source, being lack of funding for security audits, and the beauty of open source, being that eventually it will be detected and removed.

arisunz@lemmy.blahaj.zone · 35 pts · 2y (1 reply)

That's the neat part, we don't!

...but, we at least can have a shot of finding them.

In the meantime, I'd advise you to keep an eye out and maybe look into threat models. As people said in this thread already, bad actors probably don't care about your personal photos.

dylanmorgan@slrpnk.net · 12 pts · 2y

Unless OP is a celebrity or politician. Or knows they have an enemy with the resources to find and exploit potential backdoors.

Petter1@lemm.ee · 31 pts · 2y

We don’t.

doom_and_gloom@lemmy.ml · 29 pts · 2y
[ removed ]
fxdave@lemmy.ml · 29 pts · 2y (2 replies)

Afaik, most phones are backdoored that can be abused using tools like "pegasus" which led to a huge indignation in Hungary. I don't belive PCs are exceptions. Intel ME is a proprietary software inside the CPU, often considered as a backdoor in Intel. AMD isn't an exception. It's even weirder that Intel produces chips with ME disabled for governments only.

ricdeh@lemmy.world · 1 pts · 2y (1 reply)

They are not produced for governments only, almost every consumer-grade CPU can have its ME disabled or at least scuttled, thanks to efforts like me_cleaner!

fxdave@lemmy.ml · 1 pts · 2y

Have you tried it?

istanbullu@lemmy.ml · 29 pts · 2y (1 reply)

I'm pretty sure most closed source software is already backdoored.

caseyweederman@lemmy.ca · 8 pts · 2y

It's a feature!

huginn@feddit.it · 21 pts · 2y (6 replies)

The main solace you can take is how quickly xz was caught: there is a lot of diverse scrutiny on it.

Deebster@programming.dev · 16 pts · 2y (5 replies)

Hmm, not really. It's only because it nerd-sniped someone who was trying to do something completely unrelated that this came to light. If that person has been less dedicated or less skilled we'd still probably be in the dark.

N0x0n@lemmy.ml · 6 pts · 2y

Call me names... But sometimes the story has far more branched backstories than they actually shed into light.

Trust nobody, not even yourself.

possiblylinux127@lemmy.zip · 2 pts · 2y (3 replies)

The thing is there are a few thousand of those people

Deebster@programming.dev · 0 pts · 2y (2 replies)

Maybe millions of potential eyes, but all of them are looking at other things! Heartbleed existed for two years before being noticed, and OpenSSL must have enormously more scrutiny than small projects like xz.

I am very pro open source and this investigation would've been virtually impossible on Windows or Mac, but the many-eyes argument always struck me as more theoretical/optimistic than realistic.

SpaceCadet@feddit.nl · 1 pts · 2y (1 reply)

Heartbleed existed for two years before being noticed

That's a different scenario. That was an inadvertently introduced bug, not a deliberately installed backdoor. So the bad guys didn't have two years to exploit it because they didn't know about it either.

It's also not new that very old bugs get discovered. Just a few years ago a 24 year old bug was discovered in the Linux kernel.

Deebster@programming.dev · 0 pts · 2y

And are bugs harder to find than carefully hidden backdoors? No-one noticed the code being added and if it hadn't have had a performance penalty then it probably wouldn't have been discovered for a very long time, if ever.

The flip side to open-source is that bad actors could have reviewed the code, discovered Heartbleed and been quietly exploiting it without anyone knowing. Government agencies and criminal groups are known to horde zero-days.

delirious_owl@discuss.online · 21 pts · 2y

Check the source or pay someone to do it.

If you're using closed source software, its best to assume it has backdoors and there's no way to check.

ouch@lemmy.world · 20 pts · 2y

Even if there are nation state level backdoors, your personal server is not a valuable enough target to risk exposing them. Just use common sense, unattended-upgrades, and don't worry too much about it.

gerdesj@lemmy.ml · 19 pts · 2y (13 replies)

I do IT security for a living. It is quite complicated but not unrealistic for you to DIY.

Do a risk assessment first off - how important is your data to you and a hostile someone else? Outputs from the risk assessment might be fixing up backups first. Think about which data might be attractive to someone else and what you do not want to lose. Your photos are probably irreplaceable and your password spreadsheet should probably be a Keepass database. This is personal stuff, work out what is important.

After you've thought about what is important, then you start to look at technologies.

Decide how you need to access your data, when off site. I'll give you a clue: VPN always until you feel proficient to expose your services directly on the internet. IPSEC or OpenVPN or whatevs.

After sorting all that out, why not look into monitoring?

possiblylinux127@lemmy.zip · 7 pts · 2y (12 replies)

Fun fact, you can use let's encrypt certs on a internal environment. All you need is a domain.

delirious_owl@discuss.online · 4 pts · 2y (10 replies)

Just be aware that its an information leakage (all your internal DNS names will be public)

amju_wolf@pawb.social · 3 pts · 2y (6 replies)

...which shouldn't be an issue in any way. For extra obscurity (and convenience) you can use wildcard certs, too.

delirious_owl@discuss.online · 0 pts · 2y (5 replies)

Are wildcard certs supported by LE yet?

amju_wolf@pawb.social · 2 pts · 2y (4 replies)

Have been for a long time. You just have to use the DNS validation. But you should do that (and it's easy) if you want to manage "internal" domains anyway.

delirious_owl@discuss.online · 1 pts · 2y (3 replies)

Oh, yeah, idk. Giving API access to a system to modify DNS is too risky. Or is there some provider you recommend with a granular API that only gives the keys permission to modify TXT and .well-known (eg so it can't change SPF TXT records or, of course, any A records, etc)

amju_wolf@pawb.social · 2 pts · 2y

What you can (and absolutely should) do is DNS delegation. On your main domain you delegate the _acme-challenge. subdomains with NS records to your DNS server that will do cert generation (and cert generation only). You probably want to run Bind there (since it has decent and fast remote access for changing records and other existing solutions). You can still split it with separate keys into different zones (I would suggest one key per certificate, and splitting certificates by where/how they will be used).

You don't even need to allow remote access beyond the DNS responses if you don't want to, and that server doesn't have anything to do with anything else in your infrastructure.

possiblylinux127@lemmy.zip · 2 pts · 2y (2 replies)

Not if you setup a internal dns

delirious_owl@discuss.online · 1 pts · 2y (1 reply)

How would that prevent this? To avoid cert errors, you must give the DNS name to let's encrypt. And let's encrypt will add it to their public CT log.

possiblylinux127@lemmy.zip · 0 pts · 2y

Sorry I though you were referring to IP leakage. Apologizes

gerdesj@lemmy.ml · 1 pts · 2y

I do use it quite a lot. The pfSense package for ACME can run scripts, which might use scp. Modern Windows boxes can run OpenSSH daemons and obviously, all Unix boxes can too. They all have systems like Task Scheduler or cron to pick up the certs and deploy them.

lordnikon@lemmy.world · 19 pts · 2y

you don't and will never will. I would recommend reading a lecture by Ken Thompson the co-creator of Unix for more details on this https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf

avidamoeba@lemmy.ca · 18 pts · 2y

We don't. That's why we use multiple layers of security. For example keeping all services accessible only via VPN and using a major OS that a lot of production workloads depend on such as Debian, Ubuntu LTS or any of the RHEL copycats. This is a huge plus of the free tier of Ubuntu Pro BTW. It's commercial level security support for $0. Using any of these OSes means that the time between a vulnerability being discovered, patched and deployed is as short as possible. Of course you have to have automatic security updates turned on, unattended-upgrades in Debian-speak.

Oha@lemmy.ohaa.xyz · 15 pts · 2y

We dont.

HumanPerson@sh.itjust.works · 13 pts · 2y

This was likely a state-sponsored attack. If your SSH isn't exposed to the internet this probably wouldn't have effected you. Also most people run stable distros like Debian on their server, and this particular vulnerability never made it to the stable branch. I would guess that most of the computers you have ever used have backdoors. Even if you run Linux (which may itself have some) you might still have a proprietary UEFI on your motherboard. Something like xz wouldn't effect you because no government really cares what's on your server. Smaller attacks can be avoided through common sense. Some people will expose services that require zero authentication to the internet. Follow basic best practices and you will probably be fine.

Edit: Also remember, google photos once flagged a picture of a child that the child's father had taken for medical reasons as abuse, so self-hosted may not be completely private or secure, but it's better than the alternative.

radiant_bloom@lemm.ee · 12 pts · 2y (7 replies)

We don’t know. However, no one cares about your personal photos ; no one will ever attempt to hack you specifically unless you’re a high value target (in which case, stop hosting your photos anywhere immediately)

The only thing that could get your photos is if an undiscovered backdoor is exploited by someone doing some sort of a mass attack. As far as I know, they’re pretty rare, because people with the means to do them generally have a specific set of people they care about (which you are unlikely to be a part of).

lemmyreader@lemmy.ml · 12 pts · 2y (1 reply)

We don’t know. However, no one cares about your personal photos ; no one will ever attempt to hack you specifically unless you’re a high value target (in which case, stop hosting your photos anywhere immediately)

To those assumptions I would say : we don't know. Personal vendettas do exist and we cannot look into the minds of individuals going crazy neither.

radiant_bloom@lemm.ee · 4 pts · 2y

That fair enough I guess, really depends on what kinda personal photos you have. I know people are worried about revenge porn, I personally think the only actual remedy is not having any porn of yourself anywhere unless it’s your job 🤷🏻‍♀️

4am@lemm.ee · 6 pts · 2y (1 reply)

no one will ever attempt to hack you

My brother in Christ, how do you think botnets get built?

HumanPerson@sh.itjust.works · 4 pts · 2y

They did say specifically. I think bothers are usually automated attacks.

lurch@sh.itjust.works · 4 pts · 2y (1 reply)

unfortunately, mass attacks happen all the time. if you ever had access to the authentication logs of an SSH server with an unfirewalled port 22 into the internet that has been running for a few months, you would see international IPs starting port scans and brute force attacks. there is always someone out there trying to hack random IPs. it's fucking wild west out there.

billgamesh@lemmy.ml · 3 pts · 2y

Yeah. I'd recommend using ssh keys and disabling password authentication whenever something is exposed to a public network

mfat@lemdro.id · 2 pts · 2y

My ssh auth logs show a lot of login attempts from chinese IPs. That prompted me to install Fail2Ban..

pr06lefs@lemmy.ml · 10 pts · 2y (3 replies)

We don't know. But if there were well known backdoors to mainstream security practices we might see more companies that depend on security shutting down, or at least shutting down their online activities. Banks, stock trading, crypto exchanges, other enterprises that handle money, where hacking would be lucrative.

taladar@sh.itjust.works · 7 pts · 2y

I don't think you need to worry about backdoors with most of those. Worry more about unfixed security holes due to an extreme emphasis on "stability" as in using old versions when fixes have already been released when it comes to anything hosted by large companies.

mfat@lemdro.id · 3 pts · 2y

There are several known instaces of crypto exchages getting hacked.

wizardbeard@lemmy.dbzer0.com · 1 pts · 2y

There's a concept of acceptable levels of risk. Companies are not going to shut down out of fear, or miss out on the business opportunities of online presence. There's money to be made.

Even with things as serious as spectre allowing full dumping of CPU and RAM contents simply by loading a website, I can't think of a single company that just said "well shit, better just die".

Serious, potentially business ending, security issues usually have a huge amount of effort when discovered put into mitigations and fixes. Mitigations are usually enough in the immediate "oh shit" phase. Defense in depth is standard practice.

hperrin@lemmy.world · 10 pts · 2y

If backdoors exist, they’re probably enough to get your data no matter where it’s stored, so self hosting should be fine. Just keep it up to date and set up regular automatic backups.

xlash123@sh.itjust.works · 9 pts · 2y

The best you can do is use OSS software that has been battle tested. Stuff like OpenSSH and OpenVPN are very unlikely to have backdoors or major vulnerabilities currently being exploited. If you don't trust something to not be vulnerable, you're best to put it behind a more robust layer of authentication and access it only by those means.

lemmyreader@lemmy.ml · 9 pts · 2y

Good question. I have asking myself the same thing as well. In case of ssh it is possible to use 2FA with a security key, which is something I'd like to put in my todo.txt

MazonnaCara89@lemmy.ml · 9 pts · 2y (1 reply)

Ah shit we are back to "Ken Thompson Compiler Hack" again

cypherpunks@lemmy.ml · 4 pts · 2y
MonkeMischief@lemmy.today · 9 pts · 2y (3 replies)

I'm not a security specialist either. I learn new things every day, but this is why my NextCloud is accessible through TailScale only and I have zero ports exposed to the outside world.

The only real convenience I lose is being able to say "check out this thing on my personal server" with a link to someone outside my network, but that's easily worked around.

redcalcium@lemmy.institute · 17 pts · 2y (1 reply)

Next: how do we know tailscale's network hasn't been backdoored?

MonkeMischief@lemmy.today · 1 pts · 2y

I figure there's a certain amount of trust you have to have in strangers for a LOT of things we use every day.

I try to be selective with where I put that trust, especially when I can't just homebrew an advanced custom solution, but I figure Tailscale is much better than attempting to just host it on my LAN with an open a port to the big scary web and hope a bot doesn't find a gap and ransomware it all lol.

3-2-1 backups and a certain bit of trust.

Because heck, even CPUs have been found with exploitable microcode. (Spectre and Meltdown?) At some point you just gotta balance "best rational protection" with not going insane, right?

Headscale mentioned here is pretty neat too, but I feel like spinning up Dockers on Proxmox and Tailscale is as much moving parts as I'm willing to manage alongside everything else in life. :)

mfat@lemdro.id · 2 pts · 2y

I think you can use Tailscale Funnels for that.

Ookami38@sh.itjust.works · 8 pts · 2y

You never know what security holes exist until they're exploited. Nothing you can really do about that. Security and convenience have always, and will always, be a trade off and a matter of personal acceptability. If you host anything, it will be potentially vulnerable, way less so if you take proper precautions. If you're not just overtly insecure, you'll probably be fine, but there's no way to say for sure.

qprimed@lemmy.ml · 7 pts · 2y

if you are self hosting and enjoy over-engineering systems... VLANS, ACLs between subnets and IDS/IPS should be part of.your thinking. separate things into zones of vulnerability / least-privilege and maintain that separation with an iron fist. this is a great rabbit hole to fall down if you have the time. however, given a skilled adversary with enough time and money, any network can be infiltrated eventually. the idea is to try to minimize the exposure when it happens.

if the above is not a part of your daily thinking, then don't worry about it too much. use a production OS like Debian stable, don't expose ports to the public internet and only allow systems that should initiate communication to the internet to actually do so (preferably only on their well known protocol ports - if possible).

possiblylinux127@lemmy.zip · 7 pts · 2y (2 replies)

That's the neat part, you don't. However if you stay up to date it is not a big deal.

Valmond@lemmy.mindoki.com · 21 pts · 2y (1 reply)

Well not too up to date as we just have witnessed 😁

possiblylinux127@lemmy.zip · 6 pts · 2y

Just use a stable system

Dagamant@lemmy.world · 6 pts · 2y

While others are saying “no one cares about your personal photos” personal info is not the target of backdoor attacks like this. It’s more likely an attempt to get access to lots of processing power for a crypto mine or botnet.

It’s best practice to have the minimum packages required to run whatever service you are running, don’t add other stuff that you won’t be using. Using a distro that is “outdated” like Debian stable can help since the packages have had more time out in the wild to be tested.

I am sure that the xz incident has raised a lot of alarms across many projects.

friend_of_satan@lemmy.world · 5 pts · 2y (2 replies)

As many have pointed out, you don't know that there is not a back door in your software.

One way to defend against such an unknown is to have a method of quickly reinstalling your system, so if you ever suspect you have been compromised you can reload your OS from scratch and reconfigure it with minimal fuss. This is one reason I recommend folks learn one of the configuration management systems like ansible or puppet, and use those to configure your Linux servers. Having config management also helps you recover anfter unexpected hardware failures.

Defense is done in layers. No one layer will protect you 100%. Build up several layers that you trust and understand.

mfat@lemdro.id · 1 pts · 2y (1 reply)

Very good points. Call me paranoid but i always fear I might fail to notice the symptoms of something nasty going on. I wish linux had a built-in, easy-to-use auditing/alerting system. I know this can be achived by experts but others have no idea what's actually going on on their machines.

friend_of_satan@lemmy.world · 2 pts · 2y

Dude, you're so not paranoid. This stuff has happened to me. I had a Wordpress blog that was hacked and the exploit was stored in the DB so even after reloading the OS I still was infected because I hadn't sanitized my database. Luckily it was just Google search viagra spam, and it was a valuable lesson.

Shameless@lemmy.world · 5 pts · 2y
[ removed ]
MonkderDritte@feddit.de · 2 pts · 2y
[ removed ]
bitman@techhub.social · 2 pts · 2y

@mfat It's the old problem about bugs. To know that a piece of software has no bugs you should be able to count them and if you could do it then should be able to locate them and make a fix. But you can't then there's no way to know there's no more undetected backdoor

Of course being open source helps a lot but there's no solver bullet

onlinepersona@programming.dev · 0 pts · 2y

How do you know what you don't know?

That's basically what you're asking. If you have an answer to that general question, it will answer your specific question.

::: spoiler Anti Commercial AI thingy CC BY-NC-SA 4.0 :::