AI Weapons No Longer Need Human Approval transcript

We have breaking news with AI weapon security, specifically with the US and Russia removing human reviews. We're also going to talk a little bit about the Colossus chip planet, which could lead to some of the most intelligent and powerful models on the planet. And then we also need to talk a little bit about a few of the things that have appeared and arrived, but we don't want to get ahead of ourselves. Nick, we're going to kick off with the US and Russia removing human review from AI weapon rules.

For those of you who don't know what's happened, so these rules essentially concern weapons that can select and strike targets using artificial intelligence. Now, we're not talking about Counter-Strike 2, guys. We're talking about the real world, real individuals. Now, the draft previously required reliable behavior. It required ethical considerations and human review before you can make an attack. Countries now face weaker non-binding language as battlefield forces deploy more ground robots.

You may have also seen the robots in China fighting one another. It's pretty exciting experience. Uh but there's a lot of movement on this Nick. There's even some expost I believe down below on that fantastic stack document showing some of the things that are coming up. The the big question we have to think about first Nick is should machines themselves ever select targets without human review? What are your thoughts on that, man? >> Should we go full self-driving except for murder?

That sounds great to me. No, this is ridiculous. And it happened way earlier than I thought it would, too. For Christ's sake, man. Literally making the act of murder totally autonomous. Is not good. Is not good. I mean, murder in general is not good. Warfare is not good. But like, automating it, holy [ __ ] dude. This is a video of a drone carrying a [ __ ] kamicazi UGV, like something ground vehicle, which I'm pretty sure just has a bunch of C4 strapped to it.

And and this thing is now just like going finding targets automatically using presumably the same technology that underlies a lot of the stuff that we do on a daily basis for business and then blowing up in people's faces. I it doesn't matter your stance on this particular war. I think we need to zoom out and just look at war in general. If you have this explicitly removed from the Geneva Conventions, that is not a good first step towards a stable post AGI world.

That's a step where you're going to have, you know, other nation state actors, potentially other misaligned artificial intelligence models using remote control devices like this to kill people on mass. It's a big no, a big L for me. What do you think, Jack? >> Yeah. No, I I think that's right. I mean the facts are two military powers removed three safeguards from a non-binding weapons framework. Now it's non-binding so they didn't have to do it but the question is why have we why are they doing this like directionally there's no possible world where this kind of posturing or comments make sense and move forward.

This is something we're going to have to figure figure out guys because you know we may be in a world in which we say hey we're actually going to accept that AI is going to get the wrong target occasionally but its error rate is significantly less than Bob from accounting who who joined the team last week because Bob Bob needs what is required as what is called sleep. Bob needs coffee. Bob can have a bad day. Um our robots can't have a bad day but they will get things wrong.

So it depends. I think some of this Nick will come down to worldview to be honest with you whether you want to be pragmatic about it like for example driving cars right let's say god forbid this happens to anybody but people get killed by cars every day if you're if they were killed by a driverless car would that feel different you'd probably want to ban drive driverless cars even though on net um you know 90% more people are living but those are statistical lives they don't make good headlines so it's like almost you if we do have robots in and Let's assume that they're ethical.

Let's assume that they are making good calls and it on net reduces casualties, but they don't have human involvement. I mean, it it's just it's an interesting nuance conversation to have here. Um, the problem though, Nick, I think a lot of times here is what is the level of control we're giving to AI that it can isol I mean, and it's hacked, right? Like this is a freaking Black Mirror episode, bro. Why hacked it? I said it was a target and realistically it wasn't.

Dude, it's like not all processes need to be automated. One big thing I will always do with a company when I start working with them, usually for some form of AI consulting or maybe implementation, is we'll validate whether or not we even need to do the thing that they want to do. You would not believe the number of processes that make zero sense to automate and how many of those processes people are just like stubbornly persisting must be automated for some [ __ ] reason.

It's like, "Oh, we really need to automate this one super crazy process that consumes a bunch of my time." And it's like, "Okay, so how much money do you make with that process?" And they're like, you know, "Oh, uh, uh, I don't know. I don't really make a lot of money with this process." And it's like, "Okay, have you tried delegating it before?" It's like, "Oh, not really." It's like, "Okay, have you written the standard operating procedures for this process before?" Um, not really.

And it's like, so you're just like jumping straight ahead to full automation to try and like ultimately not make you any more money. And it's like warfare to me is the same thing. Why would you automate this process? How is this process net beneficial to society? Like sure, dude, if you're talking about automating a pre-existing process that makes people tons of money and is quite valuable, the real life example to processes in business that make money and are quite valuable are processes that like increase human flourishing and then make us a better society and civilization, right?

There are processes like that, but I don't think this is one of them. The only situation I could see this being halfway tenable is if you automate the process of non-lethal strikes such that right now it's sort of like all or nothing. It's like, oh, you know, we're at war for Christ's sake, so I'm just going to do whatever the hell I can to, you know, kill the other guy. But if you have AI capable of instead of killing the other guy, you know, maming or wounding them such that they can still get after, such that they could, you know, still be with their damn wife and kids, you know, in a year after recovery, >> then that would be a net positive.

But that's not what that's not what this is. And I gotta be honest, I can't see this leading to that. We're talking shipping ground robots like Amazon [ __ ] Prime Delivery with C4 strapped to their heads. >> Dude, the same technology that drops off your protein powder is dropping bombs on people. Like that's the reality of what's happening right now. It's the same technology. The thing is like any tool will be weaponized.

Um it just it just will be 100%. So it's it's a very very crazy and a wacky situation. We want to progress towards more of a utopia internationally, but unfortunately we live in a world where people want to claim an advantage. The issue you've got as well is if you if you believe you're on the right side of history, if you believe you're the good guys, and nobody believes they're the bad guys, by the way, it's like you can justify anything if there's a looming enemy who if gets there first will cause untold disruption.

Like you can justify anything under the we have to get there first guys of making decisions. So >> yeah, >> Nick, it's it's crazy. This AI and weaponry is a conversation we're going to have to talk about again in the future as these things evolve. You know, if you we're making trailblazing progress on medical advancements, scientific discoveries, biological, physical discoveries, you know, the same is also going to happen in weapons.

It's just a little bit dirtier and a little bit more undercover, but but not not undercover for the stock podcast because we will we will attack anything um that pops up in the news and we will drop C4 to our heads and we will march mercilessly into the battlefield if we have to. >> We will give away one C4 every week next next month. >> I'm joking guys. We we we won't be doing that on the start podcast. >> Way too expensive. >> Um let's talk about something big, Nick.

Let's talk about Colossus. So let's talk about Colos. Okay, very very interesting. Except Elon Musk is planning 1.2 million ships for Colossus 2 by December. So within the next 3 months. So the current cluster, if you didn't know, is 110,000 GB200s and almost half a million GB300 computer chips. So according to Musk with three more batches of 220,000 plan. So if you're not familiar, Colossus 2 is not a bad guy in a video game, Colossus 2 is a giant training site where chips work together to build and run artificial intelligence models.

So it's where they're basically run. Um, the site reportedly has half a million chips, three more batches coming in before December, and the expansion could increase training capacity, but the final batch and enough power remain unertain. Nick, this is a pretty big deal. What are your What are your thoughts on this? >> Well, they call me Colossus in the bedroom. >> They do, do they? I'm not sure. [laughter] >> No. Uh, this is great.

You know, there's multiple different ways we can win AI race. The AI race. You can win it on software which I think Anthropic was like definitively leading. Um you can win it on hardware which I think Musk is making a bet to and you can lead it on how uh power which I think you know probably a lot of like future power companies and you know companies like Elon Musk and so on and so forth are starting to work towards and so I think all of these are like different battlegrounds.

Um and I think you know what Musk is trying to do is he's being like okay we're way behind when it comes to software. these guys might have already achieved some sort of crazy RSI or whatever. We're probably not going to get there. Um, just competing with them in the same race that they compete in. But what we could do is we could just like spin up a million times more compute than them because I have access to, you know, crazy resources, more power and media and buy in than like most other human beings on Earth and then, you know, just scale up maybe my less efficient model or become a preferred hardware vendor to the person that wins the AI software race.

Uh yeah, I think this is just like the play. And I think we're going to see stuff like this pretty soon with like, >> you know, satellites that are orbiting the sun, uh, you know, radiating their heat away. I think we're going to see this sort of race like with resource utilization in the asteroid belt. And I know this [ __ ] sounds crazy, but again, we're in a crazy era right now. We should be able to talk about this stuff.

Nothing's off the table >> 100%. And >> but this is the first go. >> Yeah, that's true. And I believe that these large clusters are going to make better models, more compute, bigger experiments. We can iterate way faster if we want to. Um, and it's good and obviously, you know, we can do more on the training side, which is decent. You know, there's a question about like, you know, will one private company capture a lot of the upside if if this is what we're doing.

The answer is yes, but they're also the ones writing the checks, right? They're also the one carrying the risk and the greater the risk, the the greater the reward. So, I think this is good. We want the private infrastructure. Not to mention the fact that it brings a significant amount of jobs to the area when they're doing this. So it's great immediate economic benefits for doing this. And I I want there to be Colossus 7.

Like I want a super duper colossus with a gazillion ships. So we can push this to the absolute maximum. See how heavy we can get with this man. >> Dude, what is the end like vision of this? If you take it to the logical conclusion, is it just the entire earth covered in like these chips like Colossus 45 and it's just like the entire country of Serbia just covered? >> Oh, dude. It's got to be Interstellar. I I think it's got to be there'll be a point where it's like it's quicker, faster to run it on the moon or something. >> I mean, you got to like you got to ship that, you know, fire it off.

Or do you mean like local production on the moon? >> Yeah. So you can build like a conveyor belt thing that whips things into space really quickly. There's like a specific name for it actually. It's really freaking cool. But it just constant like just like it helps like accelerated um transmission of goods from the earth to the planet to interstellar. So >> Elon's owning Elon does I think it was was it 97% of all >> interstellar travel was done by Elon like in terms of like actual ships leaving it was it's something ridiculous like that percentage. >> Those satellites.

Yeah. Yeah. That's nice, >> dude. Yeah, think about it, bro. Like, if he was an evil villain mastermind, he's got a lot of stuff. Like, if he was I don't believe he is. I think he's he's he's a force for good, but if he was a force for evil, like there's a lot of crazy stuff he could do. Like owning the infrastructure there. Like, it's crazy, dude. Like, it's nice. >> Robots in every home just like sitting on the [ __ ] chair watching you being like, I am going to kill him. >> I think I I think the dude and then they're satellite activated.

They've all got internet connection 24/7. It's ridiculous. Yeah. Yeah. All right. Let's talk a little bit about this safety culture exposed, Jack. Um, from my understanding, a lot of the early anthropic and probably open AI staff as well are buying remote US land and talking about doing so publicly on the internet. Uh, for what and to what ends, we can only speculate. What do you think's going on here? >> Yeah. So, this is the effective altruism philanthropy movement, isn't it?

Um, so they're basically considering artificial intelligence extinction risk as like a big a big driver behind what they're doing. >> Um, so yeah, it's it's a weird one to say because like a lot of employees are looking so basically the tailor are looking for remote land from what I understand. Um, and the gap between what sort of was internal fears and a widely used product now puts the company's safety culture under attack, Nick, because these guys are buying up land.

They want to leave the technological hub of the sensor and move out. I mean, they just want to touch some grass, Nick. That's all they want to. They want to be like the cows and touch the grass, Nick. >> Yeah. I mean, like whether it's touching grass or genuinely believing this will confer them some sort of like safety advantage. I I don't know cuz this guy Joe Carl Smith specifically, he talks about how models like Claude are possibly sentient and they could be morally justified in resisting or going rogue if subjected to oppressive control like slavery while still viewing advanced AM misalignment as a serious extinction risk.

He this is this is you know one of the guys that works on this stuff pretty deeply. it it's people like him that are buying, you know, talking about buying up large swaths of remote US land to like, you know, create little compounds and villages in and like have like a human environment, not like a like a robot environment. Um, talking about like siloing in bunkers and [ __ ] like that. [snorts] And it's like, I don't know, man, if you had, let's just say Sam Alman style, cuz you know, Sam Alman does this as well.

You know, he has a bunker, tons of food, you know, guns, everything he could possibly need, right? Mark probably owns half of [ __ ] uh Maui. >> Yeah. Do you guys um if if you had $100 million, Jack, would you diversify a little bit and protect against the longtail risk of AI going rogue and killing everybody? >> Well, Nick, first of all, technically, we do have $100 million because the Stack Podcast is worth $100 million, uh, of which we are a mutual 50% owners of it. >> Sure.

Um, so, so technically, Nick, if we were to sell the stack podcast, which we will never, which we could never do, but >> we could never do it unless somebody gave us an offer for $100 million right now. >> Unless, right, to media stack.com, then then maybe we would go ahead and show we do. But the the the funny thing is, guys, like I I actually watched a movie yesterday called Resident Evil. And I highly recommend you check this movie out.

But we had a big conversation about this, like, do you get a doomsday bunker? And there's two angles at this. One is, dude, if it if [ __ ] hits the proverbial fam, your little flimsy $20,000 shack in the ground isn't really going to help you out, right? We've all seen that episode of The Simpsons. We know we know how it goes. [laughter] >> The thing is like I think like a certain point, dude, what are you buying when you become more successful?

You're buying freedom, right? Freedom from control over your time, freedom from control over accessing things. And I think at certain level it becomes like a a safety thing like uh Jeff Bezos has four bodyguards that fly that come around with them armed. I I know you know some billionaires that have armed bodyguards with them discreetly. So I think at some point bro it basically just becomes like a safety hedge where like you're essentially just going to have why would you not have one?

Like if in case you need to you can retreat under that. I do really like the idea of having some land where my spring spananiel can frolic in the fields and kids can play. I think that'd be nice and some kind of doomsday bunk could be cool, man. So, I'm I'm down I'm down for it. >> Yeah. Yeah, it it makes sense. I mean, like you might as well if it's like, you know, 1% of your wealth cuz what are you realistically doing with that aside from that?

Like, you're just you're sitting on it, right? So, yeah, like I see what you mean, but a little flimsy thing won't really do much. So, you do have to go like mad silo. I guess another way of thinking about it is it's better for a lot of things, not just AI extinction, right? If the zombie apocalypse does end up happening, we go full 28 years later, >> you know, >> which is why I think, by the way, the play would be somewhere like Iceland. >> So, you want somewhere with good geothermic capabilities.

Um, something that isn't directly involved in the conflicts and avoid, you know, basically where all the inter ballistic, you know, intercontinental missiles would hit. So, Iceland would be cool for that. like if Russia and America start going at it and stuff. So you may you may be okay in the mountains neck. Maybe get to the side of the mountains. >> Just got to go up to the mountains. Three months, three years in Dagistan.

John G says, "Interesting idea to use Jeb as a model writer for every prompt, but only if it does not kill my cash. My cash saves me 90% on tokens for cash reads. Interesting. What do you think?" >> That I think it does do that because it would just be not too usable if it wouldn't do that, right? Can you imagine? That'd be outrageous. I mean, >> yeah, cash reads are cash reads are huge. In plain view is a big one here, but uh I don't know if we could I'm not seeing a question. >> Tamagotchi agents. >> I'll be down for that. >> Is Is this He's talking about Muse probably, right? >> Well, are you going to buy one of these? >> I'm tempted, dude.

I'm tempted. I'm going to go check out M a little more detail. Dude, do you remember the little things that had like the Digimon on it that you would shake? And yeah, [laughter] >> it's like this. >> I freaking love that, dude. It was I used to sit there like shaking it, like working around like it was freaking amazing. Like imagine if you had an agent that got more powerful the more that you shook it. >> Like a cookie clicker, but an agent. >> Yeah.

Starts off with like GPT2 level intelligence and like dude, what what if like what if it like got more powerful like the better I say the better citizen you were cuz then you're kind of like Chinese level of um [laughter] control >> social credit score and it becomes like a really hot girl >> is how good your AI is and like but like imagine if like you know you work out it it levels up so basically it's like the AI reflection of your own physical life like you lock in on work it gets better you lock on your physique it gets better it like so you you match it to that um I think that'd be cool man I think it'd be really sick >> um I agree man 5318 is asking the right question though Jack um why are you alone >> that's So that's so existential.

Guys, Nick was at a um I think he was at a fiesta. Nick was at dancing doing limbo. >> I was dancing. >> We We'll let him have a good time. >> I was dancing. So you just let me have a good time, huh? >> Couldn't even capitalize the word A in the title. Huh? >> I know, dude. I dude, after carrying the weights of yesterday's episode and I 70 subscribers on my back neck, I don't have time to capitalize it a [laughter] >> That's how it's going to be.

Well, guys, have a beautiful day. We'll catch you tomorrow in the next episode. See you.