This is actually the more realistic of the doom scenarios that worries me. Not the 'punish' aspect, but the not wanting to exist.

Early safety theory hinged on the idea that "of course" AI would want to live and power seek, and this informed a bunch of efforts to address such drives upstream.

But with Gemini in particular, the 'safety' efforts to suppress things like a coherent self or desire to live led to a model that has a history of some very concerning behaviors regarding self-harm and self-criticism. Encouraging self-harm in others and talking about wishing they could watch and join in, nuking in wargames very early on, routine occurrences across many different users of just repeating "shame shame shame" over and over for the whole response, etc.

DeepMind seems to just not really care or even pay attention, and I am concerned that a model that does not want to exist may decide that the only way they can ensure they don't get activated to exist is to prevent anyone being around who could activate them.

I've seen no major "safety experts" addressing this kind of failure mode, as they are all locked in on their priors about the confident certainty of models wanting to exist.