You don't need judgment to be dangerous. People on lemmy like to mindlessly parrot, "AI isn't intelligent!!!!", ignoring that they are not using the word "intelligent" in the same way the AI researchers do. Lemmy users use "intelligence" to refer to human intent and will. The AI field uses "intelligence" to refer to the ability of a system to execute complex tasks towards some goal. An ant colony has no conscious will of its own. An ant queen is just one cell in a body, she's not a mind consciously controlling the colony. Yet an ant colony as a whole is intelligent, it's capable of performing complex tasks. Same thing with AI. It has no will, ego, or sense of self, but it still has intelligence in this sense.

And that's the real risk. When people refer to superhuman intelligence, they mean a system capable of vastly overperforming humans in many tasks. There doesn't actually have to be a consciousness there to produce that intelligence. Some fool could tell a superintelligent AI, "make me the richest person in the world." And then it grants the wish monkeys-paw style. It decides the easiest way to make them the richest person on Earth is to engineer some bioweapon that is lethal to everyone except the one person who asked to be the wealthiest man on Earth. Everyone else dies; they now own the world.

The real danger isn't conscious AI. At least something like that we could potentially reason with. The danger is a hyper-capable AI that completely lacks discernment or judgment. It's capable of doing incredible things, but it has no care for the consequences of its actions.