Selfhosted

Let's tinker around and accidentally break something.

Have you tried introducing unnecessary complexity?

If it’s stable, it’s not a lab.

That’s infrastructure.

When's the last time you checked if your backup solution works?

logging is probably down

You do, of course have a dedicated rsyslogd server? An isolated system to which logs are sent, so that if someone compromises another one of your systems, they can't wipe traces of that compromise from those systems?

Oh. You don't. Well, that's okay. Not every lab can be complete. That Raspberry Pi over there in the corner isn't actually doing anything, but it's probably happy where it is. You know, being off, not doing anything.

Now try migrating all your docker containers to podman.

Time to start documenting it!

You have remote power management set up for the systems in your homelab, right? A server set up that you can reach to power-cycle other servers, so that if they wedge in some unusable state and you can't be physically there, you can still reboot them? A managed/smart PDU or something like that? Something like one of these guys?

Oh. You don't. Well, that's probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you're physically away from home.

Have you already tried implementing an identity provider like Authentik, so you can add OIDC and ldap for all your services, while you are the only one that’s using them? 🤔

All of those systems in your homelab...they aren't all pulling down their updates multiple times over your network link, right? You're making use of a network-wide cache? For Debian-family systems, something like Apt-Cacher NG?

Oh. You're not. Well, that's probably okay. I mean, not everyone can have their environment optimized to minimize network traffic.

The rare moment when everything actually works. 😄

The comments in this thread have collectively created thousands of person-hours worth of work for us all...

Don't worry, you're one Docker pull away from having to look up how to manually migrate Postgres databases within running containers!

(Looks at my PaperlessNGX container still down. Still irritated.)

Backups. You're forgetting them.

Actually, one thing I want to do is switch from services being on a subdomain to services being on a path.

immich.myserver.com -> myserver.com/immich
jellyfin.myserver.com -> myserver.com/jellyfin

I'm getting tired of having to update DNS records every time I want to add a new service.

I guess the tricky part will be making sure the services support this kind of routing...

I should do some breaking network changes... While tunneled in.

You have an intrusion detection system set up, right? A server watching your network's traffic, looking for signs that systems on your network have been compromised, and to warn you? Snort or something like that?

Oh. You don't. Well, that's probably okay. I mean, probably nothing on your network has been compromised. And probably nothing in the future will be.

Never run:

docker compose pull
docker compose down
docker compose up -d

Right before the end of your day. Ask me how I know 😂

You can always configure your vim further

Going into spring/summer that's ideal, I wanna go places do things. Mid winter, I'm feature creeping till something breaks.

All of your systems are set up, but are they capable of being redeployed using a configuration management software package? Ansible or something like that?

Oh. They're not. Well, that's probably okay. I mean, you could probably go manually reproduce configurations, more or less.

Can't believe nobody here mentioned nixOS so far? How about moving all of your configs in a flake and manage all of your systems with it?

You have all your devices attached to a console server with a serial port console set up on the serial port, and if they support accessing the BIOS via a serial console, that enabled so that you can access that remotely, right? Either a dedicated hardware console server, or some server on your network with a multiport serial card or a USB to multiport serial adapter or something like that, right? So that if networking fails on one of those other devices, you can fire up minicom or similar on the serial console server and get into the device and fix whatever's broken?

Oh, you don't. Well, that's probably okay. I mean, you probably won't lose networking on those devices.

Then it turns out your monitoring system failed and FUCK IT'S BEEN A MONTH SINCE THE LAST PROPER BACKUP