Well, even 10 percent is 10 percent more than ideal. I think it’s fair to say that these really serious fears about AI went mainstream last week when an AI researcher and mathematician named Jacob Coxon left Anthropic. But at the same time, it’s not that big of a secret that people in the AI world think AI might kill us. Why do you think people are suddenly paying attention to this issue?
That’s a really good question. I think there are two aspects to that. For example, there’s also the aspect of why people in the industry are lagging behind at this point. As you say, they’ve always talked about it. That’s important to remember. This is not what they just said. Many other researchers have published essays or longer posts on social media addressing their concerns as well. So the industry is behind it. And people are paying attention, but it’s too late.
I think people are paying attention to AI for two reasons, not because it makes things better in their daily lives. One is hacking by autonomous agents. Similar to the hack that occurred at Hugging Face, a major AI company, it was carried out by an AI agent that escaped from OpenAI’s servers and independently hacked this other company with the purpose of cheating on tests. And then there are the incredible advances AI has made in solving mathematical problems that ordinary people don’t understand, but that even mathematicians who understand the problems recognize as important. Recently, an AI model solved a Millennium Prize math problem, which is a real breakthrough.
And I think those two things together make it suddenly seem real. So, on the one hand, the AI is out of control. They are acting independently. they can’t be trusted. They are covering their tracks. they are working together. They use strangely emotional language to explain their own decisions. They just look crazy and unpredictable. And at the same time, the models are really smart, and I don’t even like the term “superintelligence,” I don’t like the term “AGI,” but they’re doing things that I can’t do and most of us can’t do.
And the combination of these things makes this discourse about the dangers of AI, which often sounds like science fiction and sometimes even like marketing hype or something, sound really plausible and remarkable.
I’d like to go back to my first warning. The idea is that all of this could happen within 10 years. A period of 10 years is both very close and very far away. One of the lessons of climate change is that people can accept threats intellectually, but they don’t take action because they don’t feel the threats are sufficiently present. For example, New York may be underwater in 2036. It’s really hard to imagine where you’ll be or who you’ll be in 2036. Do you think the same problem will arise with AI? We can’t really do anything about this abstract threat because it’s near and yet far away, and we can’t even really imagine what it is like.
I think this will be very helpful in disentangling two concerns that are often confused. And since they’re both equally worrying, unpacking them doesn’t make them any less scary.
The first is about something called AI takeover. The idea is that AI will become very smart. They, for reasons that make sense to them, in their strange ways, decide to do things that we don’t want them to do, and those things can have really negative consequences for us. And people worry about it. They worry about AI systems that are very good at hacking, and they want their company or their country to win, so they take drastic or dangerous steps that no one else wants. Those are Skynet-type concerns. This relates to an idea often referred to as superintelligence. The idea is that AI will only become very smart very quickly, and then it will no longer be of any use to us. And I think that’s worth worrying about. And, like Bernie Sanders and Greg Cassar, he has legislation in Congress that would ban the pursuit of superintelligence activities.
But there are also more common ways in which AI is currently dangerous. In other words, the situation is already alarming. Anthropic published a report in September that looked at some of the ways people use AI and sought to stop them from using it in those ways. That includes a group in Yemen that is trying to vibrate code software for guided missiles. This includes people using autonomous bots to launch cyberattacks, much like the ones that carried out the Hugging Face hack. Therefore, it is not an abstract and distant danger. The fact is, right now, the technology that exists can be misused by people and do things that they don’t want. That’s the technology that exists today, and it’s relatively recent. But of course, it’s constantly improving. So the question is, “Will we reach a point where just one step of improvement becomes so difficult to control here and now, that we become completely disconnected from the larger, more abstract science fiction scenarios that we need to be concerned about?”
There we presented two different scenarios. AIs will become god-like beings that will go berserk and try to kill us for reasons we don’t even understand, and humans will misuse these powerful tools to create things that can harm other humans. Is it correct to say that there is also a third option, which is more of a paperclip problem, or the idea that we give an AI a task and then try to perform that task in a way that ultimately harms us all? Or does it fall into one of the other two categories I mentioned earlier? How worried are we about an uncoordinated AI that accidentally kills us all?
Well, that could happen too. In other words, all of these involve human agency and serve as a key element. It all has to do with people using these systems, which, as we said, are very powerful and at the same time very unpredictable, in ways that link them to dangerous capabilities. And the thing about AI is that it’s a pervasive technology. In other words, it is not something that can be contained like nuclear weapons. There is a free version. And the free, open source, and unregulated versions are constantly improving slightly behind the expensive and premium versions.