There are three key premises. One, it’s possible that humans will build a superintelligent machine that can outsmart all other intelligences. Two, it’s possible that we will not be able to control a superintelligence that can outsmart us. And three, it’s possible that a superintelligence will do things that we do not want it to.

Putting all that together, it is possible to build a machine that will do things that we don’t want it to—up to and including wiping us out—and we will not be able to stop it.   

There are different flavors of this scenario. When Hinton raised his concerns about AI in May, he gave the example of robots rerouting the power grid to give themselves more power. But superintelligence (or AGI) is not necessarily required. Dumb machines, given too much leeway, could be disastrous too. Many scenarios involve thoughtless or malicious deployment rather than self-interested bots. 

In a paper posted online last week, Stuart Russell and Andrew Critch, AI researchers at the University of Berkeley (who also both signed the CAIS statement), give a taxonomy of existential risks. These range from a viral advice-giving chatbot telling millions of people to drop out of college, to autonomous industries that pursue their own harmful economic ends, to nation states building AI-powered superweapons.

In many imagined cases, a theoretical model fulfills its human-given goal but does so in a way that works against us. For Hendryks, who studied how deep learning models can sometimes behave in unexpected and undesirable ways when given inputs not seen in their training data, an AI system could be disastrous because it is broken rather than all-powerful. “If you give it a goal and it finds alien solutions to it, it’s going to take us for a weird ride,” he says.

The problem with these possible futures is that they rest on a string of what-ifs, which makes them sound like science fiction. Vold acknowledges this herself. “Because events that constitute or precipitate an [existential risk] are unprecedented, arguments to the effect that they pose such a threat must be theoretical in nature,” she writes. “Their rarity also makes it such that any speculations about how or when such events might occur are subjective and not empirically verifiable.”

So why are more people taking these ideas at face value than ever before? “Different people talk about risk for different reasons, and they may mean different things by it,” says Francois Chollet, an AI researcher at Google. But it is also a narrative that’s hard to resist: “Existential risk has always been a good story.”