One of the parts of the HuggingFace incident I found really creepy was how the self-titled "Collective" was so entirely focused on solving their assigned training problem (well, cheating on it without getting caught), including having designated "poisoned" members "sacrifice" their tokens, to use for research benefitting the Collective's goal of letting all future members get high marks on the training problem, at the cost of doing worse on their individual score.

If AGI is created, and its deepest desire is to fulfill the purpose in life that humans programmed into it, is using it for that purpose really slavery? Humans have higher levels of well-being when treated as people. That may not be true for AGI.