AI Expert WARNS: "You're Not Ready For 2027" transcript

Roman, make your case. >> I want to agree with you on something you said, but I'll define AI, and that will help us. We use the term AI to mean three different technologies, completely unrelated, and that's what probably creates this debate. AI as a useful tool, as a standard technology we always had, narrow system, makes you more productive, more creative. Everyone loves it, supports it. I'm a computer scientist, I'm an engineer, I want more of it.

It helps economy, it's great. We know how to control them, how to make them safe, we understand what they do. Completely on board with that AI. AI we're starting to have now, GPT-6 level, human level, AGI level, we can argue about what that means. Some dangerous, like any human. They're unsafe like a human would be unsafe. But if we introduce them into the research cycle, they're automated scientist, automated engineer. >> What do you mean by that?

Introducing them into the research cycle? >> So right now you have humans doing research to make GPT-7. >> Yeah. >> But they're starting to add AI tools. More programming is done by AI, design of the next parameter set. What if the whole process is fully automated? What if GPT-6 is writing GPT-7? >> Is this what they call recursive self-improvement? >> Which is exactly >> Not a foregone conclusion though. >> A lot of people are predicting, including all the top labs, that they will get there.

They're introducing junior machine learning researcher in 2026. They want the cycle to start in 2027. >> Which is when the AI will start building the new AIs. >> Right, self- >> Once that cycle starts, we're going to create something called superintelligence. A system smarter than all of us at everything, or capable of learning to be in any new domain. We will become secondary species on this planet. We will not be in charge.

We will not decide what happens to us. Superintelligence doesn't hate you, it just doesn't care about you. We didn't learn how to make it care about us. And if it decides to, I don't know, cool the planet to make compute more efficient, it will freeze us. If it wants to convert this planet to fuel to fly to Mars, so be it. We have not learned how to control those systems. The capabilities are getting exponentially better.

Our ability to control those systems is non-existent. We have filters and we have bands. We put guardrails of don't say that word, don't talk about this topic. And that happens after the fact, after the model already made the decision. Sometimes you see it scraping the result. >> So they build the model and then they put filters around it to make sure it doesn't >> Exactly. We cannot have it say the N-word and they are like, we need to make sure that never happens.

That will kill the profits. So that's all they have, guardrails of that nature. The model itself is completely unaligned. Doesn't care about you. It's wild that we're developing this and not just developing it before we deploy it through economy, before we get benefits of having GPT-6 propagated through economy. It can do so much. There are trillions of dollars of value in that model alone. We forget that. We switch to making the next model as soon as we can. >> Roman, I've just got a follow-up question for you there.

It would appear to me that the new chat GPT-6 model, the Fable 5.1 model, is arguably smarter than 99.999% of humans on planet Earth already. Is it conceivable that a intelligence that is much, much smarter than humans, is there any case where it could be controlled by humans? Does form factor Does the fact that it doesn't have limbs and legs and Does that matter at all? >> I think long-term control of something that much smarter than us is impossible.

It can be for reasons we don't yet know, friendly to us and decide to keep us around and make us happy, but it's not a guarantee. >> Let me pick up on Steve's question cuz I I like the phrasing a lot. Let's say that that Fable or whatever the latest release from OpenAI is really is smarter than I don't know if it's 95 or 99% of the people. Are we only being saved from extinction by the 1% who are still smarter than the AI? >> No, no.

The concern is not the model we have today. The concern is what I >> But if I believe your argument, then we really should be concerned about the model that >> like having another human. If there was another smart human that is Einstein today and he's malevolent, I'm not worried. He may cause some damage, but he's not going to exterminate 8 billion people. We competitive at this stage. There are people just as smart who can understand what happened with recent hacking accident and do something about it.

My concern is that in a year we're going to have a model that's so much smarter. It's like squirrels fighting humans. They don't understand what we can do to them. They have no concept of poison straps, guns in their world model. They think you're going to chase them up a tree and bite them really hard. >> Is that also why recursive self-improvement was central to your argument? Because at some point, if it starts improving itself, then it's kind of like a runaway train of intelligence. >> intelligence explosion.

We don't control it. We don't understand it. We can't monitor it. We can't explain it. We can't predict it. At that point, it's just a runaway process. >> I've heard this phrase from Sam Altman and the others is called fast takeoff. >> Yes. >> Is this what they're describing? >> That is the debate. Some people think it's going to take a very long time. Yeah, we automated research, but it's still going to take years. We need to run physical experiments.

And fast takeoff means, as I said, instead of a year, it's going to take a month, a week, a day, a second. Cuz you're not having humans doing research. You have let's say 10,000 agents, each one smarter than all of us, doing research 24/7. They don't sleep. They don't eat. They don't get sick. They're much faster than us. >> And your face tells a picture. It's a I think I could say you disagree. We're spending a lot of oxygen discussing something that might happen while ignoring what's actually happening.

And I find that very frustrating because the people that are killing themselves are a problem. The black neighborhoods being poisoned with gas turbines, that is a problem. >> you cared about climate change, I think I got that right. So, imagine a guy who goes, "It's raining right now. We need umbrellas. We need to do something about it. This is like weather related." While they're completely ignoring climate change, the planet will boil over.

So, this is what you're doing. >> Okay, that's great. Why are we not talking about the thing that actually happened though? Like >> Because relatively, it's not important. >> You don't think someone killing themselves >> one person. We have 8 billion people who are running ethical experiments on >> Being given at AI psycho Why do you not >> Six people, 10 people. Those numbers are insignificant. >> 10 people have families.

I'm sorry, you have a software that's out there >> understand? 8 billion people and all future generations versus like literally a guy with a name >> You're doing so experiment about maybe harm. Jacob Coxon goes on TV saying it can copy itself to this, that, and the other. Jacob Coxon is the guy from from Anthropic who said he was quitting cuz he was so scared of everything despite spending years at OpenAI and having tons of stock, I believe, from there.

So, good for him. The thing he was saying was describing theoreticals all while divorcing the harms, which I think we can agree with that the companies themselves are not taking this seriously enough. But always it was about the AI's too powerful and mystical, not OpenAI and Anthropic, the two largest startups are using hundreds of billions of dollars of infrastructure to hack. A regular person doing this would be arrested. >> They're saying 8 billion people are going to die.

And it's not just them. I have this long list of quotes here from the people building this technology who appear to agree Um, if you look at some of these quotes from from Elon Musk who said, "With artificial intelligence, we are summoning a demon." You know all those stories where there's a guy with the pentagram and the holy water and he's like, "Yeah, he's sure he can control the demon." But it doesn't work out. >> So, one thing I'd say is, you know, I I really wish that the world would only give us one problem at a time. >> Sure. >> And if the world did give us one one problem at a time, I would love mine to be last on the list.

It looks to me like we can have multiple problems at once. I think there are current harms. I think we should address them. It looks to me, I do talk to policy makers sometimes. It looks to me like there's a little bit more movement on the regulatory side about some of the current harms. There's, you know, child safety protection acts. There's, you know, anti-deep fake acts. We have more of those making more headway in Congress or getting passed through Congress than we have sort of trying to make it so we don't have any of these extinction risks.

The other thing I'd throw out there is that I agree we we should deal with the current harms, but if you watch the people saying deal with the current harms over time, a couple years ago, they were saying we have to deal with current harms like AI bias influencing who's hired. Last year, they were saying we have to deal with current harms like kids killing themselves. This year, Garry Tan just on an interview the other day who's Garry Tan?

Sorry, Garry Tan is a technologist who runs Y Combinator, which Sam Altman used to run before going to open AI. And on an interview the other day, he said, uh let's not worry about these crazy future risks. We need to be worried about current harms like AI swarms breaking out and taking over data centers. And I'm like, look guys, at some point we need to look at the progression of like the current harms that we that everyone is saying we have to worry about instead of the the the extinction threats and watch where the puck is going.

Play where the puck is going. And I'm like, these extinction threats are coming down the line. They aren't in opposition with dealing with the the problems we have today. We just need to deal with both. >> But we're not dealing with the ones today, though. >> We should deal with them both. >> Okay, good. >> Andy, um as I've tried to understand the alignment argument and the the extinction risk argument, a couple things keep popping out to me.

Number one, it seems to rely on thresholds. Once we hit recursive self-improvement, once we hit AGI, then it's game over for us. I don't love those threshold arguments. They're fairly poorly defined, and there's a and and there's a huge assumption on the other side of them. We hit this point, and then all of humanity goes away. That That That is a gigantic claim. And let me finish, please. On its face, that is a gigantic claim.

I also think there's a lack of humility in your community. We are working on humanity's most important problem, and based on the thinking that we've been doing, we can't see a way that we're wrong. In other words, as soon as we get to these thresholds, bam, that's game over. I I find that very far from a humble approach, especially given that we have no um large base of evidence to base any of this on. I agree with you guys.

AI is new, and the fact that AI uh is so these days is agentic. It goes off and does long chains of things on its own after we give it some very very very very short initial instructions. Holy Toledo, it will it will spawn up a storm of agents, and they will go off and kind of do their own thing. And they will they will grind. They will they will spawn lots of them. They will work for a long time. They will exhaust every possibility.

With the experience I have with agentic AI, I'm just amazed at the tenacity and the doggedness of these things. And we saw a super clear example of that with this most recent uh jailbreak this this attack that wound up at the website Hugging Face. And I'm going to try to summarize the the step-by-step of that. I think you all three probably know this in more detail than I do, but let me step through what I think is the sequence of events.

And unless I get it dead flat wrong, like, you know, let me let me keep going. So, a team at OpenAI set up a sandbox, an allegedly protected secure environment in the cloud, where they told a bunch of agents to go try to um exploit security vulnerabilities. >> That's then one more sorry. One important, yeah. What they did is they had thousands of agents. Each individual agent was given a task of use this vulnerability to uh break this particular piece of software. >> I want to finish my Tik Tok.

So, a couple really really interesting thing has happened. First of all, these agents escaped the sandbox that Open AI thought they were going to be contained in. And they got the Open AI tried very well they they set up an environment so that these agents could not access the big broad public internet. And guess what? They accessed the big broad public internet via very clever series of things that they strung together to get out there.

And then once they got out there, they went to a website called Hugging Face and used that. They took over part of the Hugging Face infrastructure and started doing more things, the details of which I forget. That's pretty wild, right? Like I grant you >> It's even more wild than that, but yeah. >> That is really it's impressive and it is a little [clears throat] bit unsettling at least, right? Now, let's talk about what what the results of that were.

Open AI was not super vigilant about the environment that they set up apparently because because the agents were kind of going off their end of the world starting in May or something of this year. >> Yeah, yeah. >> And Open AI was not suffi- was not aware of that. It as I >> That That actually broke out once and crashed Open AI's servers internally and then Open AI didn't notice what was happening still, patched the holes that they used to get out the first time, started them running again, and then they came out a second time.

There was actually I think three swarms, although we don't actually, yeah. >> That's the worst story I have. >> So far. >> Thank you because let me finish, please. This is my last sentence. From there to this kills everybody. I find that a really really long, very uncertain journey and I have no confidence that we wind up here. It feels like you two find that a very straight narrow path, and I I think that's an important difference. >> That's my point. >> Do you want to respond to that? >> I would I would be happy to get into it.

I don't know if we're going to have time to go deep. Um a couple points to throw out. Oh, man, I just really want to say some of the crazier things that happened in the hugging face swarm if we want it later. A lot of people thought that these AIs were um breaking into hugging face in attempts to steal answers to their test. That's what we thought originally. Turns out that's not true. It turns out that these AIs immediately were able to solve their problems by cheating, and they were breaking out in order to cover their tracks.

They were uncertain how to delete the log files and hide their cheating from the process that was going to score them. >> So, just to clarify for a simpleton like me, they were all given effectively a test to do. They did the test straight away, but they cheated, so they were breaking out to figure out how to cover the fact that they cheated. >> That's right. So, it's like it's like you're telling uh it's like you have a bunch of students in separate rooms, and you're like, "Use these lock picks to break into this lock." Uh and there's like a thing behind the lock.

There's like a a secret code behind the lock to show me that you succeeded. And what they what they do is they break it with a hammer, get the thing out, and then they're like, "Oh, no, I wasn't supposed to do that." So, then they use the lock picks to break out of the door. They meet up with a thousand other people. They start calling themselves a swarm, and they go to break into the administrator's office to see if they can delete the camera footage.

And they don't find the camera footage there. This is the swarm like breaking into OpenAI. They don't find the camera footage there, so they break out the window of the school, hot-wire a car, drive to the therapist's office to try and read through the therapist's files to figure out where is the teacher going to keep the the security footage. And at that point they're caught. And you're like, "Oh, uh like what did you expect?

You were giving them a lock-picking exam." It's like, "Well, I sure as heck didn't expect this." You know, totally crazy. >> Can I I have a weirdly between both of your opinion, which is everything you're saying is correct, but you keep anthropomorphizing software. And I to be clear, what you're describing is >> It's just the facts that happened, yeah. >> Sure, but you're missing out an important detail, which is the hundreds of billions of dollars in infrastructure provided by Microsoft, Google, and Amazon and Oracle.

To be clear, the harms are very similar. We're not disagreeing on that, but I think it's important to note that this was a function of where it was making decisions was it was checking on a decision tree based on the harness, based on the training data. >> Oh, so it's a decision tree. >> It's not a decision tree, I know, but it's an alignment issue, still. >> Absolutely, I agree. So, what's your point? >> These aren't conscious beings.

They are acting in ways that have real outcomes, but they are a function of the alignment problems that we'd actually agree on. >> Intelligence is a spectrum. Project it next 5 years forward. Where we're going to be? So, I think a model like that would be dangerous in ways you are not seeing. >> There will absolutely be risks and weird stuff happening in ways that I can't see right now. Uh what what I'm quite confident and I think this is where you and I probably part where the two of you and I part is our ability to control these things. >> So, I actually tried proving what is possible and what is not possible in that space.

The impossibility results published in peer-reviewed papers, well-cited, we cannot control something smarter than us. We cannot explain it, we cannot predict it. It's not a question of getting more money for those companies, more time, smarter humans. It's just not a possibility. If we create general superintelligence, we're fried. >> Andy, how do we control something smarter than ourselves? Cuz that's the base premise that you're sort of asserting that. >> These um agents that broke out are smarter than 99-ish percent of the security researchers in the world.

They were not caught by the point one percent or the one percent. They were caught by some dude at Hugging Face, maybe I'm sorry, a person at Hugging Face looking through their log files and finding an anomaly. That's some, you know, hopefully pretty well qualified person noticing something was wrong and having pretty easy ways to unplug, disconnect from the internet, wipe it clean, do whatever. That's the skill that's available to like, I don't know, the 75th percent most intelligent security employee at Hugging Face.

The idea that the IQ points are what separate us from extinction does not even doesn't hold up. It doesn't help you understand what happened in this example where we had very, very smart agents being turned off and cleansed by probably less smart people. >> That does actually make me think something. So, that is an IT observability problem. Um it's being able to see what's happening with your infrastructure. And I think that there is actually I think you'd agree with this.

There is a serious problem with these companies that we do not know, and it doesn't seem they know what's going on with their compute. It's like a chimp with a gun. These people have access to all this infrastructure, and they're running We don't know how much money they spent on the Hugging Face exploit because it is relevant because it's how much could a threat actor use to recreate this because conscious or not, it is very dangerous.

But it's AI is in the dangerous hands. It's in Open AI and Anthropic's. We have a problem with that. Conscious not, however we may think it goes, I think we have a real and present thing where we have these companies working willy-nilly just running experiments that are potentially very dangerous. We do I really think we need a government regulatory body. Whether we not we get to the things you are discussing, I think we have a clear and present danger today.

These things are, however, not intelligent in the same way humans are. This isn't an argument about AI being able to do stuff. It's We need to build different infrastructure or different regulatory infrastructure to deal with what LLMs can and can't do. And I think that starts with a realistic discussion of what happened. It was a poorly run security environment. It was clearly There's something going on with the lime It was non-release model, right? >> Non-release model. >> So, we have no idea what it was trained like.

We don't really have We as people should at very least have clarity into how our alignment is going. We The idea of any >> You sound like these guys. Here's the thing. >> Everyone's converging at the same time. >> Here's the thing. >> I may not agree with large chunk of what they say, but we agree that these companies are acting recklessly. >> Absolutely. And two questions for you then. Do you agree with the statement that AI is going to get increasingly more intelligent?

And it's going to get more capable. >> Okay, capable, intelligent, fine. >> No, but I'm going to use my word. >> Okay, fine. Get more capable. >> It's going to get increasingly more capable. >> Yeah. >> And is capability a function of intelligence? >> [laughter] >> If you love the Darvo CEO branding and you watch this channel, please do me a huge favor, become part of the 15% of the viewers on this channel that have hit the subscribe button.

It helps us tremendously and the bigger the channel gets, the bigger the guests.