20 years ago. That should have been in the headline.
this post was submitted on 21 Jul 2025
119 points (86.5% liked)
Sysadmin
10461 readers
1 users here now
A community dedicated to the profession of IT Systems Administration
No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world
founded 2 years ago
MODERATORS
20 years ago? Shit, that was nothing compared to modern day amazon.
I’ve been a professional programmer for 19 years this year, and I caused 3-hour data loss on one of our major apps just the other week. Shit happens. Humans make errors; even experienced ones. We should strive to learn from it.
As they say:
Biting your tongue while eating is the perfect example of how you can mess up even with decades of experience.
Biting your tongue while eating is the perfect example of how you can mess up even with decades of experience.
Should have setup a staging enviroment and tasted there first before diving in. \s
Question: is the \s a Windows version of /s?
Or as we say in networking: "You're not one of us until you caused your first major outage".
(edit: typo)
Yeah I mean, that guy learned a valuable lesson that day. Bet he never did that again.
I once deleted a whole web server that was serving like a dozen e-commerce clients. I blame digital ocean. Both cloning and deleting had you write the servers name to confirm. Come on. Jfc. The options were adjacent.
Anyways, yep. I double check every clone/delete operation that might possibly disrupt prod ever since.
Bet he never did that again.
That's why my current boss preferes typo's over bugs. The complete opposit of my former job where they always considered burning you down to the ground for a typo. The latter makes you search for fake causes and does nothing for long term quality improvement.
I believe you
Yeah I threw like 1 week of email into a black hole unknowingly.
Instead of multiple shared mailboxes for different emails they used a master shared mailbox with a bunch of DLs all pointed to it.
The DLs were just single email contact to the one master mailbox.
Well I reinstalled Azure AD Connect (no backuo config) and didn't check off the damn contacts OU.
ALL THE EMAIL WENT TO A BLACK HOLE
After fixing the issue the email was still gone. Thankfully they had an antispam proxy service that kept a copy of all email. I restored all the aliases by resending. Managed to recover sales@company.com emails and the like.
It was the most idiotic design I've ever seen. No RBAC. Everyone just had access to "the mailbox" and they had mail filters to put it in folders. They had 3k employees too. Not a mom and pop shop at all.
I feel like some C-suite complained about "too many mailboxes" and the sysadmin "fixed it".
That person should be shot. Just because you can doesn't mean you should.
You know what, one mailbox is dangerously based. The idea of 3k employees working out of a single mailbox is so entertaining to me when I dont have to support it haha. It must have caused so many outlook crashes among other issues.
Not all employees had access to it but it was sales, accounts payable and marketing... They did manufacturing but that's still too many disciplines in one mailbox.
Hehe. The manager troll was perfect.
It's one thing to be known as the guy that crashed Amazon, it's another to be known as the "Under Qualified" guy that crashed Amazon