No, Seriously. Claude Code is Starting To Get Dangerous transcript

Anthropic CEO Dario Amodei is raising a red flag about the dangers of the rapid advancement of artificial intelligence. >> AI is more autonomous than any other technology in history. >> You believe it will be smarter than all humans. >> I I believe it will reach that level, that it will be smarter than most or all humans in most or all ways. >> For centuries, the idea that technology is somehow running out of control, [music] a self-directed and self-propelling force beyond the realms of human agency, remained a fiction not any more. >> The Trump administration has forced Anthropic, one of the country's leading artificial intelligence [music] companies, to disable its newest and most powerful AI models, citing, the company says, [music] unspecified national security concerns. >> So, this honestly used to be fearmongering just to get views, you know, taking advantage of something that most of society doesn't understand, until now.

Because we just crossed the line with Claude that we cannot come back from. Three days after Anthropic dropped the most powerful model they'd ever built, US government sent one directive, and they had the entire model shut off. And this wasn't a one-off. The same government had already blacklisted the company months earlier. Meanwhile, though, Chinese labs kept releasing open models that are catching up really fast. So, three weeks later, the US lifted the restrictions and let Fable 5 back online.

And what this tells me is that this is a race, and there's no referee, there's just a finish line, and the incentives are to win. So, yeah, Claude code is starting to get dangerous, but there's a bigger reason why we should be afraid of it. Because it won't only affect governments, it will also affect everyone who's using Claude every day. So, let me show you what this is, my predictions, and the three skills that will ensure it won't affect you.

Now, the actual order from the US said no foreign nationals could access Fable or Mythos, including foreign national Anthropic employees who are working inside the United States. But, the thing is, when someone hits an API, Anthropic has no reliable way to verify that person's nationality in real time. And because the order took effect immediately, the only way to guarantee compliance was to shut both models off just for everyone.

Now, the government said the order was triggered by a jailbreak that got Fable to scan software for weaknesses and produce a demonstration of how a flaw could be exploited. But Anthropic pushed back and said that exact test found only a few minor, already known vulnerabilities. They also later said that every model they tested could reproduce that exploit demonstration, including older Claude and GPT models. So, the dispute centered on whether that specific bypass justified recalling the entire model.

But, the capability sitting underneath it was much harder to dismiss. Fable and Mythos are essentially the exact same underlying model, just with different safety layers. Fable is the public version, while Mythos has fewer safeguards and is limited to a handful of trusted defenders. Now, before this shutdown, Mythos preview had already found thousands of previously unknown vulnerabilities and developed many related exploits, often without human steering.

It was able to find a 27-year-old bug in OpenBSD and a 16-year-old bug in FFmpeg that a bunch of automated tests had hit 5 million times without catching. So, when I say Claude code is getting dangerous, I mean it literally. I mean, put that intelligence inside an agent that can read your code base, use your tools, take action, and it can find weaknesses that human experts have missed for years. And then, on top of that, figure out how to attack some of those.

Now, during the 18-day shutdown, Anthropic trained a new safety classifier that it says blocks the reported technique in more than 99% of cases. The government tested the updated safeguards, and the controls were lifted, and Fable came back. And Anthropic also agreed to give designated government partners expanded access to future models before release and to help build a shared standard for judging jailbreaks. So, Fable returned with stronger safeguards and earlier government testing.

Chinese labs and the other American companies kept moving while Fable was offline. There's no public evidence that competition caused the government to reverse its decision, but it does show why one company, or even one country, can't slow the entire field by itself. Because the incentives right now are to win the race. This race is going to show up in places that feel a lot more ordinary than an AI lab. Now, my first prediction is that government start treating AI literacy like workforce infrastructure.

They're already spending money on domestic compute and AI companies, but the money is also starting to reach classrooms and workers. The White House created an AI education task force and NSF's AI Ready America program funds AI literacy, workforce training, and small business adoption. Now, what I think is that it's going to get much bigger over the next, you know, 5 to 10 years. Because a person who loses a job may be offered a publicly funded AI credential through a community college or a workforce program, or maybe our AIS certified AI consultant program.

Teachers may get AI training as part of their normal professional development. Local business programs may pay to help small companies adopt this type of stuff. Because no country wants millions of workers waiting for AI to happen to them while a rival country is teaching all of its people how to use AI. So, AI education starts to look less like a niche like coding boot camp and more like basic computer workforce literacy.

The race reaches schools, libraries, community colleges, and workforce offices. So, that's prediction number one. My second prediction is that cybersecurity becomes one of the biggest career winners from AI. But, the job changes a little bit. Now, the Bureau of Labor Statistics already projects that information security analyst jobs are going to grow 29% from 2024 to 2034. AI agents add more code, more access, and more automated actions for those defenders to actually secure.

And AI will probably handle more of the first pass alert review and vulnerability scanning. The human job is going to move more towards the parts that carry real responsibility, like deciding whether the AI is right, controlling which agents have access to what, investigating the weird cases, responding when something goes wrong, and proving the system is actually secure. And cyber insurance is going to make this very, very real for normal businesses.

Insurers already care about things like MFA, multi-factor authentication, software updates, backups, employee trainings, vendor access, and incident response plans. The AI questions would obviously get added to that list rather than just replacing it. Because controlling your own agents doesn't stop somebody from using AI against you. Attackers can use it to make, you know, phishing emails more convincing or scan for weaknesses and move faster once they get into a system.

So, I think the renewal form eventually starts asking questions like, which AI agents can send money? Which ones can touch customer data? Can any of them deploy code? Are their actions logged? Who can shut them off? And if a company can't answer those questions, its coverage may get more expensive or harder to obtain. But, a cyber insurance policy doesn't stop the attack. It may help cover some of the damage and support the response afterward, you know, depending on the policy.

But, the business still has to protect its people, its accounts, its software, its vendors, its data, everything. And regulators can publish standards, and insurance can put a price on ignoring them. But, cyber insurance is going to become one of the gates that forces companies to prove they take the entire risk seriously. So, that's number two. Now, the third one is that governments will try to create some version of AI arms control, but I don't think we get a clean global pause where everybody agrees to slow down.

I just don't think that that's very realistic. Countries have already signed international AI agreements, and the United Nations is studying how frontier AI claims could be verified. Training a frontier model from scratch requires massive clusters of specialized chips. You need some serious power, you need some, you know, data center infrastructure, and a huge amount of money. A person in a basement, realistically, I don't think they could do that right now.

But, that same person may be able to download open-source model and fine-tune it and remove safeguards or use it in ways the original developer never intended. This stuff is great. AI is really intelligent, but it's an amplifier of people's intentions, and that idea is a little scary sometimes. Control gets much harder after the copy starts spreading. So, the first AI arms control system will probably be a messy stack of chip tracking, cloud logs, model evaluations, incident reports, [music] data center inspections, things like that.

And that could make the biggest training runs more visible, but it cannot prove that every lab, every country, every private actor, every person in their basement will actually stop. And the incentives to cheat are enormous. The same governments that may want stronger safety rules are also funding domestic compute. You know, they're training their workers, and they're trying to avoid dependence on arrival countries' models.

And I have to show you guys this because I've never ever seen a Twitter thread go this viral this quickly ever. So, this guy, Jacob Coxen, he tweeted this yesterday. I resigned from Anthropic today. I spent the last 3 years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

So, I'm not going to read this whole thing, but he goes on this whole tangent and each of these have millions and millions of views. But basically, do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and real resources. Progress is not slowing. Now, what happened after this tweet? Lots of people quoted this.

Bernie Sanders said, "Mr. Coxen is right. The very people building this technology admit that it could threaten the future of humanity. This is why I will soon be introducing legislation to ban superintelligence and pause AI development. But what happens if that only actually goes into effect in the United States?" Here we have another one from Evan Hubinger, which if I real quick look at his profile, he's the alignment science lead at Anthropic, previously at OpenAI and Google.

But anyways, he said, "Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think that it is greater than a 10% chance within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are clearly not on track to. I mean, come on guys, this is a really serious topic. Every lab knows that slowing down alone could mean losing the lead or losing the race.

The fable shutdown was a small preview of that problem. One government could shut off one model for 18 days, but it could not shut off the capability, the competition, or the pressure to keep moving. And that leaves normal people and businesses with a pretty weird problem. We can't stop the race, and avoiding AI altogether probably isn't realistic either because, you know, the person using it well is going to move way faster than the person who isn't.

So, the goal is to use all of that capability without handing over your judgment, your access, or your responsibility. And that comes down to three skills, which are to find the pain, control the system, and to own the outcome. Okay, so skill number one is to find the pain. A stakeholder, a client, whatever you want to call them, someone may come to you and say, "Hey, you know, I saw this AI agent on LinkedIn. Can you build that for us?" And if you just open Claude and build exactly what they asked for, you may end up automating something that was never the real problem.

The more powerful these models get, the more important it is to figure out what problem actually deserves that power. I'll give you a quick example to make this stick. I had a client once who came to me who owned a med spa, and she came to me because she wanted AI systems that would bring more leads through the door. Because the pain she thought she was feeling was she wanted more revenue flowing through the business, right?

And before we built anything, I asked one of my favorite questions, which is if your business grew 10x tomorrow, which process would break first? And once we actually looked at that business, getting leads was not the immediate problem. They already had tons of leads coming in, but a lot of those leads weren't showing up to their appointments that they had booked. So, the med spa was leaking revenue after the lead had already been generated.

So, putting more leads into that pipe would have just made that leak more expensive. So, the better first project was actually improving the follow-up and reducing no-shows, because that directly attacked the constraint that was already costing her business money. And this is where your judgment becomes really, really valuable. Claude can execute the wrong plan insanely fast. You still have to understand the business well enough to point it in the right direction.

And that value starts before the prompt ever gets written. But once you've picked the right job, you have to decide how much of the business the agent is actually allowed to touch. And that is skill number two, which is controlling the system. You wouldn't hire somebody on Monday and immediately hand them, you know, the company bank login or an entire database. But businesses are kind of doing the digital version of that when they connect an AI agent to everything at once: their email, their CRM, their payment system, their code base, all of this kind of stuff without really thinking through permissions.

An agent does need access to do useful work, yes, but access isn't all or nothing. It's not binary. You can decide what it can read, what it can change, what actions require [music] approval, and whether you can trace exactly what it did afterward. So, I would always start with like read-only access whenever you can, have the agent only do drafts before it pushes anything into production or sends anything to a client, and always keep a human approval step around things like money or deletions or publishing or anything in production.

And then make sure you're logging all the actions, and make sure you also can shut the whole thing off quickly if it starts doing something weird. And this is basically what the agent-specific part of the cyber insurance prediction looks like at the business level. An insurer may eventually ask which agents can touch customer data, who approves the high-risk actions, you know, where the logs are stored, and what happens if one of these agents gets compromised, things like that.

If you can answer those questions, you're way more prepared for the risks created by your own agents. So, just remember that more capable agents need tighter boundaries, and even with a really good set of boundaries, an agent can still produce a polished answer that's completely wrong. The last skill is owning whatever comes out the other side of these agents. Because AI can draft the email, it can write the code, it can respond to a customer, it can make a recommendation, but it can't take responsibility when it quotes the wrong price or emails the wrong person.

You are the one who has to own that. So, what you do is before the build, you decide which business number this system is supposed to change. Maybe it's getting more customers or making each one worth more or cutting costs or preventing expensive mistakes. And the more specific you can be about that, the better. Then, what you're going to do is you're going to write down the baseline number, the target number, and the amount of time that you're giving the system to get there.

But, I would add one more question now, which is what could this system get wrong, and who's responsible for checking it? So, for the med spa, we would have tracked the appointment show-up rate before touching anything. We would have set a target and measured that same number after the follow-up system went live. But, because the AI is communicating with real customers, we'd also need to review what it's saying. We need to decide when a human takes over the conversation and keep a log of every single message and every single interaction that it has with an actual client.

Now, when the stakeholder or the business owner or whoever it is asks you, "Did this thing work?" you can show exactly which number changed, and if they ask, "Was this under control?" you can show how it behaved, who approved it, and what happened when it wasn't sure. That proof becomes more valuable as these systems get more powerful. Because businesses are going to need people who can produce the result and stand behind it.

So, the whole idea is, if your agents make mistakes, you are responsible for that and you have to own that. But, on the other side of things, if your agents do things that are really good, you should get credit for it, and you need to make sure that the business understands that. As Claude keeps getting more capable, your role kind of moves up a level because you need to decide what should be automated, you control what the system can touch, and you take responsibility for what comes out the other side.

Now, I know we covered a lot lot information today. So, what I did is I put all of this into a free resource guide for you, completely for free, inside of my community. And you can find that link down in the description. But, anyways, that is going to do it for this one. So, if you guys enjoyed, or you learned something new, then please give it a like. It helps me out a ton. And as always, I appreciate you guys making it to the end of the video.

And I'll see you on the next one. Thanks, everyone.