The important thing to keep in mind is that we are not currently at risk of anyone inventing a super intelligence. The word guessing engines are getting amazingly good at guessing words, but the huge jump from that to actual intelligence, super or otherwise, remains out of reach with currently known technology.

I agree that if we did manage to invent that, the biggest threat would be the humans doing short-sighted, greedy and generally stupid shit with it. No coincidence that that's also the biggest threat from current LLMs.

Edit: "Super-hammers could break containment and kill us all if we don't invest more in controlling them!" shouts man wildly swinging around a hammer in a crowded room. "The government needs to regulate this technology so that only people like me, who use it responsibly, can have access to it," he added. He tried to say more, but was out of breath after some particularly enthusiastic hammer swings.

The good news is that instead of research that might lead to something that is meaningfully super intelligent, or even intelligent in any moral agent sort of way, instead we've put all the AI funding into LLMs and transformer architectures. There's likely little to no research with any odds of actually progressing in that direction because of LLMs instead sucking up all the oxygen. There's nothing to worry about in the near future, and anyone telling you otherwise has something to sell you, or really doesn't know what they're talking about.

If something happens that causes us to actually need to grapple with the idea of a sentient machine in the next 50-70 years, society will have gone on a very different course than it seems to be, and we'll all be very old if we're still around.