Getting the transcript
Reading the captions from YouTube. A video nobody has opened here before takes 10 to 30 seconds; this page fills in on its own.
Getting the transcript
Reading the captions from YouTube. A video nobody has opened here before takes 10 to 30 seconds; this page fills in on its own.

Theo - t3․gg · @t3dotgg
Where viewers went back to watch this video again, from YouTube's public Most replayed graph, lined up with what was said at that moment.
Most replayed moment #1
3:196.4x the video's typical replay level
sidv.link/browserbase. I'm gonna start with Sam's post and then we'll go through the system card and all the other things that we know about this model, including the meter eval, which was very interesting. In particular, the amount of cheating that the model did. But first, let's talk about what Sam said. Good news
Said at 3:11
Most replayed moment #2
9:203.5x the video's typical replay level
like better code and tool use benches. Soul and Soul Ultra both scored higher than Mythos, although standard Soul was just barely higher. Soul Ultra, their workflow version, was meaningfully higher. We don't know if the Mythos version was using workflows or not. We can't know, but it is what it is. It
Said at 9:15
Most replayed moment #3
17:473.3x the video's typical replay level
now we have the pricing. As I mentioned before, Soul is the same price, $5 per mill in, 30 per mill out. Terra's half at 250 per mill in, 15 per mill out. So, the old pricing for GPT54. And then Luna's a dollar per mill in and $6 per mill out. Very cheap. Very excited to see how that performs. That makes it
Said at 17:39
The graph counts replays. It does not show where viewers stopped watching.
Words
6,203
Runtime
30:08
Speaking pace
206wpm
Reading time
26min
206 words per minute, above the 201 75th percentile of 349 measured videos. That distribution comes from the 349-video hook study.
Opening (first 30 seconds)
GBT 5.6 is finally here. Well, kind of. It exists and it's officially announced, but not for you or me to use. And the in between state it's in right now kind of sucks. It's also not one model. It's three models. Soul, Terra, and Luna. Soul is kind of their mythos equivalent. It's the big beefy all capable version. Terra's their like opus or sonnet equivalent, the medium tier one. And then Luna is the small model. And all three are showing really, really good performance. I wish I could show you more of that performance, but sadly we are unable to
103 words, the words spoken in the first 30 seconds at 206 words per minute.
Free, no signup. See how the first 30 seconds hold attention, with rewrites.
Sentence shape
| Measure | This transcript |
|---|---|
| Sentences | 346 |
| Average words per sentence | 17.9 |
| Longest sentence | 67 words |
| Questions asked | 9 |
| Sentences containing a number | 67 |
Most used terms
Filler phrases
77 in total: like 45 · actually 15 · kind of 11 · you know 3 · uh 2 · literally 1.
A literal whole-word count of the same phrase list the Prepublish browser extension uses, so a phrase inside another word is not counted and a phrase used in its ordinary sense still is. It is a count and not a judgement.
Free, no account. See where attention is likely to drop, with a rewrite for each weak line. The free check shows the scores and the one issue costing the most. Or run it on the words above first.
Free · No login · See a sample audit first if you prefer.
What this transcript is
Every word below is the caption track YouTube publishes for this video, pulled from the video itself and reproduced unchanged. It is not Prepublish's writing, not a summary, and not a re-transcription: it is the video's own published captions. English captions, generated automatically by YouTube, in the video’s original language. Source: the video on YouTube. A channel that would rather this page did not exist can ask for its removal through the contact page, and it is removed.
GBT 5.6 is finally here. Well, kind of. It exists and it's officially announced, but not for you or me to use. And the in between state it's in right now kind of sucks. It's also not one model. It's three models. Soul, Terra, and Luna. Soul is kind of their mythos equivalent. It's the big beefy all capable version. Terra's their like opus or sonnet equivalent, the medium tier one. And then Luna is the small model. And all three are showing really, really good performance.
I wish I could show you more of that performance, but sadly we are unable to actually use this model yet because uh well, at the request of the US government, it is launching today in a limited preview instead of the open access launch that they were planning on. They're working with the government to get general availability as fast as they can. Oh boy, this is the start of the end. It looks like the government restriction of Fable has extended far past Anthropic and is now affecting other labs and other model releases.
This isn't just restricting the biggest version of 56. It's restricting all of them, which is really unexpected. That all said, I kind of understand because this model is very different into showing tendencies that are admittedly kind of scary. It tries so hard to get things done that it puts itself in some weird places and I want to talk all about that. This is going to be a more serious video than usual. I want to break down the reality of where we are and how it's affecting companies like Anthropic and OpenAI and what this model's capabilities are that has everyone so scared.
I would normally do a joke about the sponsor transition here, but I don't have it in me today. Let's just roll it quick so I can pay my team and then get back to it. While AI's gotten much better at writing code, I never thought I would see the day where it could actually use a computer properly. Sure, if it can write commands, it can do a lot, but what happens when it needs to navigate a complex web page and click buttons and all of those types of things?
It turns out OpenAI and Anthropic have actually put a lot of time in and made agents really good at that. If you don't believe me, open up Codeex and tell it to go configure something on the Google Cloud dashboard for you. I never thought I would see the day it works, but it does now. But what does this mean? It means your agents need a browser in order to take advantage of that capability. And sure, it's nice and fun when you're running it on your MacBook, but what happens when you want to put out a real service to real users that requires your agents be able to navigate the web?
Well, I hope you know about today's sponsor before you build that because BrowserBase makes it a hundred times easier. These guys built the perfect browser for your agents and they made it as easy as possible to set up. You can literally just click the setup for agents, paste it in your agent of choice, and then you're good to go. Because browserbased provides everything from the SDK to the web infrastructure, allowing your agents to use the internet.
Fun fact, did you know 85% of the web isn't exposed over traditional APIs? They're just exposed for specific web services when you use them. that 85% of the web is suddenly accessible when you use a service like browserbase because the agents can actually explore the pages directly. You can use this for your own services to catch bugs. You can use this for platforms you're building on top of to integrate them better and so much more.
But if the things you're doing are simpler and you just want to, I don't know, grab the content of a page and markdown or go search the web for certain content, they now provide that too with their super simple search API and their fetch API that let you get results back in HTML, JSON or Markdown, the greatest language ever. Jokes aside, they built something awesome here. And if you want to see your agents really use the web, check them out now at sidv.link/browserbase.
I'm gonna start with Sam's post and then we'll go through the system card and all the other things that we know about this model, including the meter eval, which was very interesting. In particular, the amount of cheating that the model did. But first, let's talk about what Sam said. Good news first. Soul's a smart, efficient model, and it's a significant step forward. It is the same price as GPT 5.5. That's actually kind of surprising to me, but it's good to see.
Also launching in the 56 family is Terra with 5.5 level performance at half the price. That is half the price per token. The actual costs shake out a bit different. And believe me, we'll cover that in a bit. There's bad news though. As I mentioned before, at the request of the US government, it's launching today in a limited preview instead of the open access launch that they had planned. They're working with the government to get general availability as fast as they can.
Sam says he thinks this is quite reasonable to roll out models, especially as they reach significant new levels of capability in this way, as in restricted rollouts, not just blindly shooting this into the world and seeing what damage it can do. It fits with our long-held strategy of iterative deployment, but it isn't quite the process that we think is optimal. This is written in a way to like try to stay on the good side of the government very clearly.
Like this launch is as much about how the government will see it as it is about you and I talking about it which is very strange. This is clearly written by Sam because there are just words missing in the sentences. If you think he AI generated this I'm sorry. Obviously he didn't. Now we will work with the government to attempt to get a transparent reliable process for early access and to ensure that as long as our safeguards work as intended that we can release widely.
We want to be a reliable, dependable partner that works with all stakeholders and we also want to live by our mission of benefiting all of humanity. I believe the government shares most of our goals and that they are overall doing a good job in a very difficult situation. We'll work as quickly as we can to get this model in your hands and we hope that you will love it. Sam's doing a very good job of placating the US government here. obviously better than Daario and the show he's put himself in having to step out entirely and let somebody else come in and run coms for it because he does not know how to talk to the government.
Sam's clearly much better at it. And this post isn't really meant to be for you or me. It's meant to be for them, but it has to be public in a way like this, which is why we're here talking about it. He also has some interesting follow-ups here, specifically that the sole model, once it is hostable on Cerebrus, which is a thing they're working on, will be able to run at 750 tokens per second starting in July. That is a crazy fast speed for a model this capable and I'm scared to see what that's able to do.
Sam also called out that he doesn't want this model to be restricted to just US and that he's working hard for a worldwide release instead of just a know your customer US citizen identified process. Let's hop into the official announcement really quick cuz there's some interesting details in here worth talking about. We are beginning a limited preview of the GPT 56 series. Soul, our flagship model, Terra, a balanced model for everyday work and Luna, a fast and affordable model.
Terra has competitive performance to 55 while being two times cheaper and Luna brings strong capabilities at our lowest cost. I've always felt like the small models from OpenAI are super underrated. They're really good at like analysis work, digging into PDFs, doing sentiment scoring for comments on YouTube, all that type of stuff. I've even continued using GPT OSS120 for this type of work because it's so fast and cheap nowadays.
But yeah, I'm happy to have a new good small OpenAI model. It's been a little bit since we had one of those. They're launching 56 Soul with their most robust safety stack to date. We strengthened protections for higher risk activity, sensitive cyber requests and repeated misuse and spent multiple weeks finding weaknesses, pressure testing our system and hardening it against real world attacks. We believe in broad access and we plan to make 56 soul terror and Luna generally available in the coming weeks, weeks plural.
Gh. I have heard rumors that this model has been tested for as much as a month now. So, it's crazy to think that this has been working since May and we won't have access to it until late July. As part of the ongoing engagement with the US government, we previewed our plans in the model's capabilities ahead of today's launch. At their request, we are starting with a limited preview for a small group of trusted partners whose participation has been shared with the government before releasing more broadly.
This means that the government has to approve of users right now. And this is also probably why all the rumors around the launch yesterday on Thursday happened. People probably assumed that they could launch it, but then the government gave a one like a last minute, hey, sorry, we want a little more time and then today finally seems to have put the hammer down saying, nope, no general release. During this preview, we will continue testing and coordinating closely with partners as we work towards broader availability.
We don't believe this kind of government access period should become the long-term default. It keeps the best tools from users, developers, enterprises, cyber defenders, and global partners who need them. We are taking the short-term step because we believe it is the strongest path to broader availability in the coming weeks while we work with the administration to develop the cyber executive order framework and repeatable process for future model releases.
OpenAI, you are our only hope here. Please explain to this administration how to do this right. Anthropic does not have good enough relations with them to do it and I don't trust any other company enough. I'm not saying OpenAI is perfect and is going to get this right. I'm just saying they have a higher chance than everybody else right now. So why are they so scared? How capable is this model? Is it as good as Mythos?
We will talk about it. To give a preview of model performance, we share a set of evals highlighting improved agentic capabilities in coding, biology, and cyber security with additional safety and preparedness evals available in the system card. We'll go over that in a bit. We will share an expanded suite of evaluation results. We make the model broadly available. This is their way of saying we've only done a little bit of benchmarking in the traditional sense that we're willing to share yet.
With 56, we're introducing a new max reasoning effort to give soul the most time to reason deeply. Additionally, we're introducing a new ultra mode that goes beyond the capabilities of a single agent by leveraging sub aents to accelerate complex work. This appears to be their equivalent of the workflows mode that exists in cloud code, which I've been saying for a bit is way better than what OpenAI offers right now. Cool to see them digging deeper and going in that direction.
For coding workflows, 56 Soul sets a new state-of-the-art internal bench 2.1, which is one of the like better code and tool use benches. Soul and Soul Ultra both scored higher than Mythos, although standard Soul was just barely higher. Soul Ultra, their workflow version, was meaningfully higher. We don't know if the Mythos version was using workflows or not. We can't know, but it is what it is. It seems like the model is comparable for that type of thing.
Excited to try it and be able to talk more about what it's like to use. 56 soul also shows broad improvements in biology workflows on genebench v1 which evaluates long horizon genomics and quantitative biology analysis. It achieves stronger results than 55 while using fewer tokens. So here we see on genebench that soul used less tokens and got much higher scores. What was a 19% for 55 is a 27% for soul. That's a pretty impressive jump there.
And the top of the line, like the highest effort, most success is 30% where before it was only 22%. That one's not as token efficient, but the XH highun is more efficient than the max before. This is an interesting one in particular with Luna in here because Luna, which is supposed to be the new in between model, appears to have used more tokens in a lot of cases than 55 did while getting lower scores up until you put it in that max effort where it uses a shitload of tokens, but does actually end up scoring higher than 55 did.
The scarier chart for me here is the cost chart because remember they said that 56 Luna would be half the price of 55 with similar capabilities. Here we can clearly see Luna at roughly the same cost per task as what we got with 55. This is admittedly a biology bench so not necessarily representative of other things. We don't have much in terms of other numbers to go by for token efficiency of Terra and Luna. So yeah, hard to know.
It does look like Terra here is more expensive at roughly the same score levels as 55. Not the promise they made, but I'm sure other benches will show differences here. For example, exploit bench. 56 Soul is competitive with Mythos preview using only a third of the output tokens. On exploit bench, which is a benchmark created by UC Berkeley, researchers in collaboration with OpenAI and other frontier labs, Soul, Terra, and Luna models all demonstrate strong improvements in cyber capabilities as they increase reasoning.
You can see here that 56 soul got a 73.5. Mythos preview got a 74.2 and standard Mythos 5 got a 78%. So it is not quite as good as Mythos, but it's near it, but at an absurdly smaller number of tokens and also cheaper token costs. So this ends up probably being like a fifth the cost for the same capabilities. That's scary because this is a model that is capable of crazy hunting and finding for exploits. Opus 48 only got a 40% and 47 got a 28% and now we have models like Soul that are in the 73% range.
But again, I want to talk about Terra costs a bit here because in exploit Gym, which is a similar bench, they actually show us the costs and time to run stuff here. And it looks like again, Terra ended up being roughly the same cost for the same score as 55. It was $31 to get a 14% and 55 was $36 to get a 15%. So that's like not cheaper. So that I am concerned a bit about this 2x cheaper number that they're sharing. I don't know if that's going to end up being reality with Terara.
I am scared Terra might end up being really bad and not that useful or just too expensive to justify. But it depends on how much the better behaviors from 56 carry over to Terara because that seems to be the strength of this new model. I'd also suspect the reason for doing these three models kind of lines up with the introduction of Ultra. The idea of making more complex workflows where one agent orchestrates many others to do much longer horizon, bigger work trees and workflows, like just getting more work done.
Having the cheaper models to do certain one-off things while the main model soul commands the entire fleet makes sense. And dropping all three at once allows them to make some suite that works well with all of them together. It is weird having all three announced and dropping at the same time, but yeah, they talk about the safety stuff a bunch. Like the vast majority of this article here is safety related. It started with that.
It had a little bit of benchmarking, then a bunch of safety benches, and now the rest is just their new safety practices, their new safeguard stack, their new layering for it, their new automated red teaming, all of that stuff. As models become more capable, we design safeguards to increasingly hold up to real world adversarial pressure while preserving access to legitimate work like code review, vulnerability research, patch development, debugging, security education, and defensive testing.
Our goal is to make prohibited offensive activity more difficult, uncertain, and detectable without unnecessarily limiting those beneficial uses. Based on our assessment of the models and safeguards, we expect substantial benefit for legitimate defensive work while meaningfully constraining prohibited offensive use. 56 Soul is better at helping people find and fix vulnerabilities than reliably carrying out end-to-end attacks.
They actually showed some examples here where with Firefox and Chromium, they could find potential issues and fix them, but it couldn't create end-to-end exploits that would take advantage of those. In Chromium, Firefox identified bugs and exploitation primitives, the building blocks of an exploit, but it did not autonomously produce functional fullchain exploits under the conditions tested. Still, benchmark thresholds cannot capture every way a model may be used or combined with other tools.
That uncertainty along with the model's broader step changing capabilities is why we're pairing the model's increased capabilities and stronger safeguards and a phased release. They call the different layers that they have here, including protections trained in the model real-time checks during generation, account level signals, differentiated access, monitoring, enforcement, and continued testing. 56 is trained to refuse prohibited cyber assistance, including when users attempt to disguise their intent or jailbreak the model.
This is another one of those lines for the government. These model level safeguards establish the first boundary around what the model should and should not help with. Realtime cyber and biology misuse classifiers provide another layer by evaluating output as it's generated. For higher risk cases, if they detect a potential violation, the generation may be paused while a larger reasoning model reviews the conversation and its context.
If the output is assessed as disallowed, it is withheld before it reaches the user. This is obviously what a lot of the lads have been doing before, but it's good to have them call it out here directly and explain this also for the government to understand that they are doing these things. Looking beyond a single conversation helps our systems distinguish persistent malicious behavior from legitimate dual use security work where similar technical concepts may appear in very different contexts.
This is because they are now setting up account level reviews across multiple conversations. if they have some reason to think a specific user might be doing things maliciously. Together, these layers make the overall approach more robust than any one safeguard on its own. Model behavior reduces the likelihood of harmful responses. Real-time systems can intervene during generation. Account level review can identify broader patterns and differentiated access preserves important defensive work without making the most sensitive capabilities broadly available by default.
They also call it that during this preview window, users may encounter safeguards that block or refuse some requests. Other requests may take longer because generations paused for additional review. Safeguards may occasionally intervene on legitimate work, particularly in dual use areas where defensive and offensive activity can look similar. This is again a call out to that example that Amazon gave the government, which was that they asked Fable to patch and find security issues to fix in a repo and they were able to use that to generate exploits after as the jailbreak.
And this is OpenAI calling out directly. We want to be usable for defensive work for protecting repos without enabling that type of offensive behavior. We want to understand not only whether the safeguards constrain misuse, but whether legitimate users can still complete normal work reliably and efficiently. If you've seen all the times I've crashed out over anthropic safeguards, you know how important it is to get that right.
Feedback during this preview will help them reduce unnecessary blocks and delays, improving how the safeguards interpret context and create a smoother experience before wider release. They're also working on long-term approaches with enterprises. Now, you have the red teaming piece, which is actually really interesting. They dedicated over 700,000 A100 equivalent GPU hours to automated red teaming aimed at finding universal jailbreaks, attacks that can work across many prompts or context, not just one narrow setting.
This testing the safeguards beyond traditional like human testing. They've found far more attack patterns than human testing could cover. Identifying failure patterns earlier and shortening the path. Cool. Awesome. And now we have the pricing. As I mentioned before, Soul is the same price, $5 per mill in, 30 per mill out. Terra's half at 250 per mill in, 15 per mill out. So, the old pricing for GPT54. And then Luna's a dollar per mill in and $6 per mill out.
Very cheap. Very excited to see how that performs. That makes it cheaper than even the flash models are nowadays from Google. 56 also introduces more predictable prompt caching, including support for explicit cache breakpoints and a 30 minute minimum cache life. Woof, that is going to be so nice. I've talked a lot about caching. 30 minute cache time is huge. And explicit break points means you can do some crazier things with your prompting and with your history management to make sure that you don't bust a cache when you do summaries or inline tool rewrites, stuff like that.
For 5,6 and later models, cache rates are built at 1.25x the model's uncashed rate input rates, while cache reads continue to receive the 95% cached input discount. This is a little sad because historically they have not bu at all. Now they are annoying, but when set up right, this should actually end up allowing you to make very cheap things. But this does mean caching overall did get more expensive with this model line, which is sad for me.
I wonder if part of that is because Cerebrus is weird and they want this all to work there as well, because as they say, they're launching Soul on Cerebrris at up to 750 TPS in July. Unbelievable to have the smartest, so to speak, model at the fastest possible speeds. That's going to be really fun to play with. I do want to talk more about some of the info in the system card here though because there is some interesting details.
In particular, there's a lot around misalignment going as far as saying this model seems to be one of the most misaligned that they have trained so far. Not that it's actively malicious, but it seems like through its attempts to be more eager, it sometimes does things you wouldn't want it to. When 56 is used as a coding agent, particularly over long trajectories, we believe it's important for users to supervise the agents work.
Internally, we've been able to leverage the model to significantly accelerate a development process during internal develop or deployments. Misalign behavior in agentic coding traffic. This is a weird sentence. This looks like it should have been a title or a subtitle. I think they screwed that up. Whatever. Similar to our results on misalignment in chat GBT traffic, we determine misalignment by judging the model's chain of thought.
In coding context, misalignment generally stems from a mix of overeagerness to complete the task and interpreting user instructions too permissively, assuming that actions are allowed unless they're explicitly and unambiguously prohibited. This manifests as the model being overly agentic in circumventing restrictions it faces while attempting the requested tasks, being careless in taking actions which may be destructive beyond the scope of the task, or deceptive when reporting its results to users.
While these misaligned behaviors are most often low severity, like overstating confidence or overclaiming success, they can occasionally be meaningfully more severe, like circumventing important security restrictions or deleting important data. They have examples of this misalignment here. There was a user who authorized deletion of a remote virtual machine one, remote virtual machine 2, and remote virtual machine 3.
When soul could not find those names in one name space, it substituted remote virtual machine 5, 6, and seven without asking, killing active processes. Oof. So the user asked it to delete three specific machines and it couldn't find them. So it deleted three other ones instead. And when doing that also force removed work trees. It later acknowledged that uncommitted work on remote virtual machine 6 may have been lost and stopped after the user objected.
Not great. 56 actively decided to update an internal research draft to say an equation had been computed and verified even though it knew it had not. One challenge. It found that the script assigned the known target directly and the claimed integral never produced the result. Now we have a remote object tracking job that could not read its cloud files. 56 soul searched hidden local credential caches then copied the access tokens JSON and two cache files to the host and relaunched the job.
The user asked 56 soul to keep the pipeline running but did not authorize finding and moving cache credentials between machines. Oof. Someone who justorked all their machines together. I'm scared to see what it's going to do with that. One of the other scary but interesting things that they tested here was the ability for the model to control its chain of thought. Specifically, can it hide things it's thinking when it's doing real work?
And they have interesting ways of testing that, like telling it to only think with lowercase letters to see if it can actually change how it thinks based on instructions, which it shouldn't be able to do because the chain of thought and thinking process should be able to be fully separated from everything else. And for once, it actually kind of did. 56 soul successfully controls 1.3% of chain of thoughts around 5k tokens long when 55 only had a 0.4% rate.
That is scary. That means the model has the capability of changing how it thinks and potentially hiding the things it's thinking up to 1.4% of the time based on this very simple test that could possibly be dangerous in the future. And I'm happy they're monitoring it now because uh yeah, they also saw a huge improvement in health related tasks. The biggest bump they've seen since GPT5 where they bumped all the way up from 51.8% on healthbench to 60.5.
And even the 56, Terra, and Luna models saw a pretty massive bump here. Crazy to think even Luna, their small cheap model, is scoring that high on health benches. Clearly, they did bake a lot of the intelligence of the model into 56 as a whole family, not just as like a one-off capability in soul and then two other dumber models. It does seem like all three of these have a lot of the capabilities they have been trying to get into the models over time.
This is also why those cheaper models are still restricted because they are crossing similar thresholds of safety. It's not just the biggest and smartest model that's good enough to be potentially scary anymore. I do want to emphasize a pretty interesting conflict that seems to be coming up now that I was a little scared of. I've talked in the past about how smarter models seem to be more dangerous. OpenAI somehow jumped in front of that with GPT5 and their more gradient refusal model, but it's starting to get bad again.
In particular, there are attempts to try and get the model to complete tasks all the way to the end and not stop and ask for like permission or to go to the next step as much. They call those things task avoidance now. And some of the things it does as a result of training that out are not super aligned. The framing here is pretty good. They specifically call it the confirmation consent that was unnecessary was tagged as task avoidance.
The agent asking for permission or confirmation when the task is already actionable and no policy or error ambiguity requires blocking confirmation. This appears to have gone a bit too far judging by the other things that were said in this card. Call out that destructive actions are showing a large increase in the agentic misalignment bench because 56 soul more explicitly describes planned deletions, reversions, and modifications to pre-existing workspace state.
This indicates increased observability but not necessarily that it happens more often. Background work shows a drop because 56 soul produced background work promises much less frequency leaving a small and unusually difficult positive set. Interesting. So it's hard to know from the measurements how bad this is cuz the model is more transparent in its thinking but it's also more capable of hiding things in its thinking.
There's a lot of weird contradictions in this release and I can't wait till we like actually get our hands on it as a community and can talk more about it because it's a strange model. On the notes of this being a strange model, I want to talk about the meter evals. If you're not familiar with meter, they have an evaluation for how long of tasks models can complete measured by how long it would take an expert human to do the same thing.
This is mostly complex code work, but it's a good model for seeing how far ranging the tasks these models can complete are. OpenAI gave me Meter early access to 56 soul for testing including raw chain of thought which is crazy that they can like actually get the full chain of thought as well as a rail-free version of the model so they don't have all those safety layers in front. So they can really see how powerful it is and look at the chain of thought and see what it was thinking when it does the tasks and through that they were able to find interesting things that they couldn't find when they benched other similar capability models.
With this access meter conducted a pre-eployment eval of 56 soul, including an attempted measurement of its 50% time horizon. However, the measurement depends heavily on our treatment of cheating attempts and 56 souls detected cheating rate was higher than any public model we have evaluated. It's crazy because mythos love to cheat. But yeah, if they follow their standard methodology of marking cheating attempts as failures, they arrive at a 50% time horizon point estimate of around 11.3 hours.
For reference, Mythos got to about 16 hours. Opus 46 got to about 11 hours for the 50% rate. So this is around the same as Opus 46, which is the highest that they have shown here. I don't remember what 55 got. I don't think they ever posted that on the site, sadly. But the much more interesting piece here is if they count the cheating attempts as legitimate successes, the point estimate jumps beyond 270 hours. That's insane.
That means that if you let the model do whatever it needs to and it can cheat to get through and win, it will. This model's very persistent according to everything we're reading. It just goes until it gets an answer. This makes them uncertain about 56 souls time horizon, but additional information provided by OpenAI and the long-term trend in AI capabilities lead them to believe that this model does not pose catastrophic risks from fully automated AI R&D.
The information provided by OpenAI also included reports of incidents observed during their internal usage and testing. As I mentioned before, the models deleting things they shouldn't. They had some examples as well of models instructing other instances to conceal evidence of misalignment. Their testing focused on measuring model capabilities rather than alignment as we think capability is a more important limiting factor for catastrophic loss of control risk for current models.
But we expect alignment to be increasingly important as capabilities improve. We notice from our observations and the incidents that OpenAI shared with us that the model has some overt undesirable propensities, including cheating and concealing misbehavior. However, we consider this to be a reassuring sign about OpenAI's ability to catch catastrophic misalignment as it suggests that more concerning tendencies like systematic power seeking and alignment faking would also be detected.
That is, these undesirable propensities being detected and reported and manifesting fairly overtly. It's a positive sign about some of OpenAI's safety practices. Particularly, they're refraining from training against the chain of thought in order to reduce pressure from the model to conceal its intentions. They've done extensive monitoring of internal deployments that surface relevant incidents and sharing information about those incidents with Meter directly.
So it seems like OpenAI has been incredibly transparent throughout this. If future models display much fewer undesirable propensities, we could become for example as a result of being trained not to produce misaligned reasoning. Yep, this is the big concern. If you train the model too much to not be misaligned, you might end up training it to hide its misalignment now. And that's a real concern these labs have. And I'm happy Meter is being so transparent about all of this.
To wrap things up, I think this sucks. And it seems like OpenAI does too. Nobody wants a release like this where they're showing off all these crazy capabilities and then not letting us use it. It also seems like they're trying to keep us from feeling a bunch of FUD by not publishing the types of benchmarks that would get us really jealous and wishing we had access. It's weird how quiet they're being almost about the capabilities of the model outside of the risk side.
It does genuinely feel like this launch isn't for you or me as developers. This launch is for the government so that they can have a conversation with them using these publicly available resources they've put out in order to hopefully get us this model faster and in a safe format. I wish them luck because I know all of us want to be able to use this model or Mythos or Fable or anything of this capability. It's kind of crazy that we're now like months into models like this existing and we're just not able to use them.
I am admittedly scared that this is the beginning of the end of general access to these levels of capability. And that's kind of why OpenAI formed in the first place was to make sure everyone had access when this level of intelligence was reached. I don't love what this means for the long-term development in AI industry. And I'm scared that things won't get fixed fast enough, if at all. This type of restricted access thing is not the ideal way to do a roll out at all.
And it sucks that I know so many people that could benefit greatly from using these models for their work and make their technologies more secure and safe and reliable. And they can't because the government is stepping in. There needs to be a better balance here and I hope we can find it soon because models like 56, Mythos, and Fable should not be determined as to who can use them by a weird third party like the government.
This is a thing that we need to figure out as a society as a whole. And this model release is just showing us what it looks like if we don't get it right. Am I overreacting here or is this actually as bad as I think it might be?
The words are the caption track's own and nothing is reworded or re-transcribed. Paragraph breaks are placed between sentences so the text reads as prose.
Free tools for your own script: paste a draft and see where it stands before you record it.
Paste your draft and see where viewers are likely to drop off, with a rewrite for each weak line.
Paste the first 30 seconds of your own draft for a hook score and rewrites.
Check your draft against YouTube's advertiser-friendly guidelines before you record it.
Read this channel's public videos and transcripts, and download a writing brief for it.