41
PSA: unattended-upgrades restarted nginx in the middle of my stream (lemmy.world)
submitted 1 week ago by to c/selfhosted@lemmy.world
14 comments fedilink hide all child comments
 

Small thing but it got me last week. I relay my stream to twitch through nginx-rtmp on a cheap VPS, and one morning unattended-upgrades pulled a new systemd, needrestart restarted nginx (and postgres, and everything else) and the stream dropped. OBS reconnected fast but twitch counted it as two streams.

default timer is 06:00 UTC + up to 1h random, which is exactly when I stream lol. moved it with a drop-in:

# systemctl edit apt-daily-upgrade.timer
[Timer]
OnCalendar=
OnCalendar=*-*-* 13:45:00 UTC
RandomizedDelaySec=15m

the empty OnCalendar= matters, without it you just add a second time and keep the old one.

anyone else got bitten by this?

all 14 comments

sorted by: hot top controversial new old
[–] 21 points 6 days ago (5 children)

Unattended upgrade?

Why? Specially on critical systems?

If you do such upgrades, schedule for a time you don't use the system at very least ...

  • permalink
  • fedilink
  • source
  • hideshow 5 child comments
    •
  • [–] 10 points 6 days ago (2 children)

    There's few apps I can think of that fail when the service restarts. At worst Jellyfin and Home Asssistant take a minute or so to be available again. The former usually does not pose a problem because of client side caching while streaming. HA would be quite annoying, altough lights etc would keep their current state so it's unlikely I'd even notice.

    That said I don't use unattended upgrades.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 6 days ago (1 child)

    Yeah problem is when upgrades break stuff....

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 1 point 5 days ago

    it's a single cheap VPS that only I look after, so I'd rather get security patches the same day than remember to log in and do it, I guess that's the tradeoff. and yeah you're right about scheduling, that was the whole fix, the default timer runs around 6 UTC plus up to an hour of random delay and that happened to land on top of the stream. moved it to the afternoon and it's been quiet since

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 5 days ago (1 child)

    I've been running it on a Debian VPS for years as a test. So far not a single problem, although I don't let it reboot on its own (I do monthly reboots, that I am thinking of changing to fortnightly).

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 1 point 5 days ago

    same here on the reboots, I never let it reboot on its own either. for me the reboot wasn't the problem, it was needrestart bouncing nginx and postgres right after the upgrade, which you only notice if something long lived is running at that moment, like a stream lol

  • permalink
  • fedilink
  • source
  • parent
    • [–] 2 points 6 days ago* (last edited 6 days ago) (1 child)

    I mean, unattended upgrades has it's own config file where you can specify when to schedule a reboot (when needed), which you'd usually do in off-hours like 2 AM your local time, so changing the Systemd unit activation time seems like the wrong fix.

    Edit: Nevermind, your issue was that the daemons were being rebooted, not the whole system, if I understood that correctly

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] [S] 2 points 5 days ago

    yeah exactly, no reboot, it was needrestart restarting nginx and postgres after the systemd update. you're right that the timer is kind of the blunt fix, the cleaner one is probably telling needrestart to only list instead of restart ($nrconf{restart} = 'l' in needrestart.conf), or excluding nginx from it. I went with the timer because I still want things restarted, just not while I'm live

  • permalink
  • fedilink
  • source
  • parent
    • [–] 1 point 6 days ago (5 children)

    I've been burnt by this, my nginx lxc on proxmox was crashing at about 6:30 every morning for a week. Turns out auto update and debsecure was running on it and using all the 512mb of memory and the lxc would get killed by the host, so removed both as I update Tuesday evenings anyway

  • permalink
  • fedilink
  • source
  • hideshow 5 child comments
    •
  • [–] [S] 1 point 5 days ago (4 children)

    512mb and getting killed every morning at the same time sounds like a fun one to track down lol. a fixed weekly slot like your tuesday evening is probably the sanest setup honestly, I just know I'd forget.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 5 days ago (3 children)

    I have some of my works colleagues join and we all do system updates and talk crap for the evening, its a great way to help others sort things

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] [S] 0 points 5 days ago (2 children)

    that's actually a nice idea, updating with people around makes it way less of a chore, and someone always knows the weird fix. I'd probably steal it if I had more than one box to look after.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •