Selfhosted
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
view the rest of the comments
Kinda unrelated to OOP, but out of curiosity, what does your storage setup look like? Do you keep stuff reasonably backed up with that much data?
Ah yes. My storage system is 2 x Supermicro CSE-846 cases. Only one has a CPU and motherboard, the other is acting as a plain Jane JBOD.
Hard drives I have 21 x 8TB 7200RPM mix of Seagate and Western Digital and 4 x 16TB 7200RPM from Seagate. I use mergerfs and snapraid. Mergerfs presents all the 21 8TB drives as one mount point. Snapraid uses the 4 16TB drives to provide 4 parity drives. Note that snapraid is not live and the parity is only updated after running a "snapraid sync" which I run nightly.
I only backup my songs and music videos. The rest is easy to get again. I have a script that generates a list of every single file I have each night. So if the day comes it wouldn't take too long to get back to where I was. The other reason I use mergerfs is if 1 drive dies, I only lose the files on that one drive and not the entire array. The truely important stuff such as tax documents, mortgage details, family pictures, will & estate documents are stored on a 2 x 8TB RAID1 and all backed up nice a safe using Proxmox PBS. The PBS datastore is synced to 2 remote locations as well as to external drives that I keep offline and rotate.
Nice write-up. I thought I had a large library (24TB) and my off site backup is starting to get full. I backup everything though but I have long debated on if there's a point of keeping movies and TV since they'll likely always be available. Anyway, I never thought of generating a list of files and eliminating the stuff that's not particularly important. Good idea.
I used to back everything up before I broke the 50TB mark. Just can't justify it now. I even looked at LTO drives for backing up the multimedia but they're still to expensive for the higher capacity drives. And then you need tapes..
All the truely good content will always be out there somewhere on the net.
The script I use to generate the file lists is very very basic. Nothing special no formatting the lists or anything since it's just for that oh balls, everything is gone scenario.
ls -alR /mnt/volume1/media > /mnt/volume2/backups/file_lists/media.txt
ls -alR /mnt/snapraid/data* > /mnt/volume2/backups/file_lists/snapraid.txt
Those text files are also part of the files backed up with PBS so I can always go back and restore previous versions of them. You may ask why I generate the list twice? The first is just everything inside the media folder on the volume1 mount point. The second let's me see what files are on each individual drive so if only 1 drive dies I can just grep the text file and output to another text file and show me what is on that 1 drive.
Thanks for the detailed write up!