So today I discovered that there's a cron job that holds non-reproducible state that died, and now our system is fucked.
The cron job doesn't live inside any source control. This morning it entered a terminal state, and because it overwrites its state there's no way to revert it.
I'm currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.
Had a similar thing once. Some how, some way, the DBA copied and pasted something wrong. Oracle DB had some odd extra syntax for left and right joins that other DBs didn't (or at least that I'd never seen). My best guess is that he auto formatted out of habit and maybe it took those symbols out.
It took a long time to find that. Because the only evidence something was wrong was that ONE of our customers wasn't being billed for ONE product. Everyone else was fine. Basically they were using it in a very atypical way. The left joins made sure to include them in the billing even because they didn't have whatever was on the right of that join. Everyone else did.
SQL auto format is still mostly terrible
The only half decent format is to start from the Mozilla style and then make it more sane.
I’ve been playing with sqlglot lately and want to start using it for diffs.