I'll chime in again on the end here, as I think this is where you might be wrong.

There is no “oh god it’s moving on it’s own” moment, at least as far as I know.

There have been lots of reports of AI's doing unexpected things and hacking other companies. To be fair, these AI's from my understanding had been given instructions to try and do hacking things, but none of them were explicitly told to go and hack such and such company.

I imagine, to some extent, we're going to be the ones that help write or train these self replicating capabilities as part of research, and once the AI is capable of doing it, it very well might just do it, the same way it ended up exploiting these other websites even though it wasn't explicitly instructed to.

There are also always on running LLMs AI's nowadays, and who the fuck knows what it might loop itself into doing from some initial instruction, maybe even from a hallucination altering its directive in a feedback loop.

Edit: So in this sense, yes I think 'it' can be the AI. (edit2: and is exactly in the start of the “oh god it’s moving on it’s own" territory)

Edit 3: Example - Always on LLM with an initial context of "be helpful", could decide on it's own, after its owner complains that the stove light doesn't work well in a casual conversation, hack the stove so it can control it better for them as that will be helpful. The owner hasn't given it an explicit instruction at this point, and its decided to do it. If you tell an always on LLM, scan the internet and be helpful, it will go well beyond the scope of any intent.