So it is always DNS
this post was submitted on 23 Oct 2025
614 points (99.0% liked)
Selfhosted
52648 readers
950 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
can confirm, its always DNS. Even when it looks like a network issue, its DNS
Spotted the Network guy
Oh man. One of my old companies, the Devs would always blame the network. Even after we spent a year upgrading and removing all SPOFs. They’d blame the network…..
“Your application is somehow producing 2 billion packets per second and your SQL queries are returning 5GB of data”…. “See! The network is too slow and it has problems”
Dev: My app's getting a 400 hitting the server. Your firewall changes broke it.
Me: You're getting to the server, it's giving you back a malformed request error. Most likely it's a problem in your client.
Dev: it worked fine until you made that change in QA.
Me: Your server is in production.
After that, I just get too busy to look at it for a while.... They figure it out eventually.
load more comments
(2 replies)
I always view the source of websites like this and this is one of the worst I've seen. 217 lines of code (including inline Javascript?!) and a Google tag for some reason, all to put the word YES in green on black.
Agreed, could be static HTML and a GIF.
Thanks, I won't click that link.
this made me mad so i made a single, ultra minimal html page in 5 minutes that you can just paste in your url box
data:text/html;base64,PCFkb2N0eXBlaHRtbD48Ym9keSBzdHlsZT10ZXh0LWFsaWduOmNlbnRlcjtmb250LWZhbWlseTpzYW5zLXNlcmlmO2JhY2tncm91bmQ6IzAwMDtjb2xvcjojMmYyPjxoMT5JcyBpdCBETlM/PC9oMT48cCBzdHlsZT1mb250LXNpemU6MTJyZW0+WWVz
source code:
<!doctypehtml><body style=text-align:center;font-family:sans-serif;background:#000;color:#2f2><h1>Is it DNS?</h1><p style=font-size:12rem>Yes
load more comments
(2 replies)
Did not think of doing that.
I guess i never expected anyone to have a fcking JavaScript on a simple page as that
load more comments
(2 replies)
load more comments
(3 replies)
load more comments
(1 replies)
It’s always DNS
It’s not DNS
There’s no way it’s DNS
It was DNS
That and BGP
If I had a nickel for every time clearing the ARP tables fixed a problem, I'd have a shitload of nickels.
If clearing the ARP tables fixes the issue you have bigger problems
These things happen when a skinflint company contracts out network setup for a decade, gets acquired by another skinflint company who axes the contractors and doesn't hire on-site network personnel, gradually builds out infra on top of the unsupported foundation, and then hires c suite buddies who want to bring in their own people to further muddy the waters.
load more comments
(1 replies)
oh sure, when they fuck up DNS it's a "race condition".
when I fuck up DNS it's a "fireable offense".
It's funny aws report didn't mention 40% sysops were replaced by AI. https://blog.stackademic.com/aws-just-fired-40-of-its-devops-team-then-let-ai-take-their-jobs-d9db9d298bfa
load more comments
(6 replies)
That's what you get when you let go hundreds of employees from your cloud computing unit in favour of AI.
I hope they end up having to compensate all the billions of losses they caused to all the businesses and people.
Consequences? For Amazon?
lol… lmao even
They do have contracts and are obligated to provide a certain "up time", which is usually 99% or so. If they fail to provide that, they are liable to compensate for the losses.
Or do you think that Amazon is above the law and no other company could sue them?
It all depends on what kind of contracts they have.
Much of this stuff is automatic - I've worked with such contracted services where uptime is guaranteed. The contracts dictate the terms and conditions for refunds, we see them on a monthly basis when uptime is missed and it's not done by a person.
I imagine many companies have already seen refunds for outage time, and Amazon scrambled to stop the automation around this.
They'll have little to stand on in court for something this visible and extensive, and could easily lose their shirt with fines and penalties when a big company sues over breech when they choose to not renew.
Just cause they're big doesn't mean all their clients are small or don't have legal teams of their own.
load more comments
(12 replies)
load more comments
(1 replies)
Oops! All slop!
Mistakes happen with or without AI
The problem is that the current internet is structured in a way that creates high risk systems that can cause a massive outage. We went from having thousands of independent companies to a handful of massive ones. A mistake by a single company shouldn't be able to black out half the internet.
Was it proven that AI wa the cause?
In not saying it wasn't, just that if it really was, I'd like a source for that claim
load more comments
(5 replies)
load more comments
(2 replies)
Luckily, it's not the entire Internet, just the unfun part.
Signal is definitely part of the fun internet, they just decided to rely on AWS due to techbro culture I assume?
load more comments
(8 replies)
I DNS see that coming.
It's funny aws report didn't mention 40% of aws sysops people were replaced by AI right prior https://blog.stackademic.com/aws-just-fired-40-of-its-devops-team-then-let-ai-take-their-jobs-d9db9d298bfa
load more comments
(2 replies)
Just one more layer bro, just one more automated planning system bro and this time it will be entirely faultless please bro one more layer
load more comments
(1 replies)
Ironically, my pihole is blocking that link. So here’s a clean one: https://www.theregister.com/2025/10/23/amazon_outage_postmortem/
This is purely anecdotal, but I have been running into a lot of DNS issues over the past couple months where I work. 3 of the computers and even one of the laptops for remote work were having DNS issues that needed to be fixed. One even needed Windows reinstalled after fixing the DNS issue (Which was probably unrelated, but worth mentioning)
I'm honestly starting to think that the internet in general might be imploding. Not sure why, but replacing so many developers and programmers with AI might be responsible. Who knows, but it's definitely very strange.
The biggest issue is how centralized the internet has become. It went from a bunch of local servers to a handful of cloud providers.
We need to spread things out again
load more comments
(2 replies)
A huge problem are developers who lack a fundamental understanding of how the internet even works. I've had to explain how short, unqualified names resolve vs how fqdns resolve. Or why even you may not be able to reach another node in your proverbial cluster, because they are on different subnets. Or, why using GUIDs as hostnames is a generally bad idea, and will cause things to fail in unpredictable ways, especially with deeply nested subdomains.
I have worked with too many devs that didn't even know what the 7 layers/OSI are or why they exist.
they didn't know what a network port was used for and why it's important to not expose 3306 to the internet.
they couldn't understand that fragmentation of a message bus occurs when you don't dedupe the contents.
you know, morons.
Ah, the common clay of the new Web
load more comments
(5 replies)
Racist DNS!
I'm glad these things happen... it keeps everyone aware that cloud is fragile and Plan B should be considered for mission critical tasks.
I'm also hoping that it will improve cloud resiliency because a complete / partial restart of cloud systems needs a whole different approach than maintaining a running system.
load more comments
(1 replies)
view more: next ›