this post was submitted on 21 Jul 2025
121 points (86.7% liked)
Sysadmin
10461 readers
3 users here now
A community dedicated to the profession of IT Systems Administration
No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Or as we say in networking: "You're not one of us until you caused your first major outage".
(edit: typo)
Yeah I mean, that guy learned a valuable lesson that day. Bet he never did that again.
I once deleted a whole web server that was serving like a dozen e-commerce clients. I blame digital ocean. Both cloning and deleting had you write the servers name to confirm. Come on. Jfc. The options were adjacent.
Anyways, yep. I double check every clone/delete operation that might possibly disrupt prod ever since.
That's why my current boss preferes typo's over bugs. The complete opposit of my former job where they always considered burning you down to the ground for a typo. The latter makes you search for fake causes and does nothing for long term quality improvement.
I believe you
Yeah I threw like 1 week of email into a black hole unknowingly.
Instead of multiple shared mailboxes for different emails they used a master shared mailbox with a bunch of DLs all pointed to it.
The DLs were just single email contact to the one master mailbox.
Well I reinstalled Azure AD Connect (no backuo config) and didn't check off the damn contacts OU.
ALL THE EMAIL WENT TO A BLACK HOLE
After fixing the issue the email was still gone. Thankfully they had an antispam proxy service that kept a copy of all email. I restored all the aliases by resending. Managed to recover sales@company.com emails and the like.
It was the most idiotic design I've ever seen. No RBAC. Everyone just had access to "the mailbox" and they had mail filters to put it in folders. They had 3k employees too. Not a mom and pop shop at all.
I feel like some C-suite complained about "too many mailboxes" and the sysadmin "fixed it".
That person should be shot. Just because you can doesn't mean you should.
You know what, one mailbox is dangerously based. The idea of 3k employees working out of a single mailbox is so entertaining to me when I dont have to support it haha. It must have caused so many outlook crashes among other issues.
Not all employees had access to it but it was sales, accounts payable and marketing... They did manufacturing but that's still too many disciplines in one mailbox.