Sorry to hear that. Do you know how it was compromised and how you found out?

I'm confused. I never disable root user and never got hacked.

Is the issue that the app is coded in a shitty way maybe ?

I've been quite stupid with this but never really had issues. Ever since I changed the open ssh port from 22 to something else, my server is basically ignored by botnets. These days I obviously also have some other tricks like fail2ban, but it was funny how effective that was.

I can’t even figure out how to expose my services to the internet, honestly it’s probably for the best Wireguard gets the job done in the end.

I'm having the opposite problem right now. Tightend a VM down so hard that now I can't get into it.

I do worry about putting up public servers that other people might rely on because there's something I might not realize making it vulnerable.

So far I have pubkey root login only on the VPSs I'm messing around with, but my ol' reliable private key from 6 years ago might be beginning to fall behind on encryption standards.

One time, I didn’t realize I had allowed all users to log in via ssh, and I had a user “steam” whose password was just “steam”.

“Hey, why is this Valheim server running like shit?”

“Wtf is xrx?”

“Oh, it looks like it’s mining crypto. Cool. Welp, gotta nuke this whole box now.”

So anyway, now I use NixOS.

Interesting. Do you know how it got compromised?

Basic setup for me is scripted on a new system. In regards to ssh, I make sure:

• Root account is disabled, sudo only
• ssh only by keys
• sshd blocks all users but a few, via AllowUsers
• All 'default usernames' are removed, like ec2-user or ubuntu for AWS ec2 systems
• The default ssh port moved if ssh has to be exposed to the Internet. No, this doesn't make it "more secure" but damn, it reduces the script denials in my system logs, fight me.
• Services are only allowed connections by an allow list of IPs or subnets. Internal, when possible.

My systems are not "unhackable" but not low-hanging fruit, either. I assume everything I have out there can be hacked by someone SUPER determined, and have a vector of protection to mitigate backwash in case they gain full access.

At least you had a backup

Permitting inbound SSH attempts, but disallowing actual logins, is an effective strategy to identify compromised hosts in real-time.

The origin address of any login attempt is betraying it shouldn’t be trusted, and be fed into tarpits and block lists.

On a new linux install or image I will always:

• Make new users(s)
• Setup new user to sudo
• Change ssh port
• Change new user to authenticate ssh via key+password
• Disable root ssh login

And this is why every time a developer asks me for shell access to any of the deployment servers, I flat out deny the request.

Good on you for learning from your mistakes, but a perfect example for why I only let sysadmins into the systems.