AI
mp
The property that has given humans a dominant advantage over other species is not strength or speed, but intelligence. If progress in artificial intelligence continues unabated, AI systems will eventually exceed humans in general reasoning ability. A system that is “superintelligent” in the sense of being “smarter than the best human brains in practically every field” could have an enormous impact upon humanity. Just as human intelligence has allowed us to develop tools and strategies for controlling our environment, a superintelligent system would likely be capable of developing its own tools and strategies for exerting control. In light of this potential, it is essential to use caution when developing AI systems that can exceed human levels of general intelligence, or that can facilitate the creation of such systems.
Since artificial agents would not share our evolutionary history, there is no reason to expect them to be driven by human motivations such as lust for power. However, nearly all goals can be better met with more resources. This suggests that, by default, superintelligent agents would have incentives to acquire resources currently being used by humanity. (Just as artificial agents would not automatically acquire a lust for power, they would not automatically acquire a human sense of fairness, compassion, or conservatism.) Thus, most goals would put the agent at odds with human interests, giving it incentives to deceive or manipulate its human operators and resist interventions designed to change or debug its behavior.