How Google DeepMind Runs Agents at Scale — KP Sawhney & Ian Ballantyne, Google DeepMind transcript

Okay. Hello everybody. So, my name is N Ballantyne. I'm a developer relations engineer at Google DeepMind. Hi folks. I'm KP Sawney, software engineer in DeepMind's AI platform team. We're going to do an agentic panel today to talk a little bit about how DeepMind thinks about agentic software, how we build kind of our own stacks. We're going to start very briefly by just showing a quick demo and then we'll go into like a like a discussion about some of the stuff that KP works on and hopefully we can just you can ask lots of questions, find out all the things you want to know about how DeepMind and Google think about agents and yeah, let us know your thoughts as well.

So, quick show of hands, who's actually used Antigravity before as a tool? Okay, so like four or five people. Um, I just want to show you one quick thing cuz one of the things that we have in Antigravity is a lot of people know that it's like a Visual Studio style interface, but they don't actually know that it actually has a whole agent manager and agent manager framework behind it. So, you can run and spawn multiple agents working on different projects and it's integrated into the IDE, but it's also again kind of like an agentic platform in itself.

So, you can do things if I've got this project here and you can have this chat panel on the side and I can just say like build an example of this spec and I can give it a particular file. Let's do the spec file. Let's just implement that one. And I'll send that off to flash. Oh. It's experiencing errors right now. Let's restart. Should we try that? I am connected to the Wi-Fi. I'm pretty sure I am. Yes, and if you're not connected to the Wi-Fi, you should use the Gemini models instead.

That's my plug. Uh so, let's start that up again. And hopefully I can do the same thing. Build uh build the spec marked here. There you go. Let's see whether we can get it this time. Okay, there you go. So, again, the model will just go away thinking uh on the side there. And hopefully what it should do is it should have a look at the spec. It should analyze it, and then it should use a bunch of its own internal tools to decide how to do that.

Uh it's actually found that there are already existing files that I have built. It might even tell me that's already done it, uh which is interesting. But, the tool itself has like built-in to-dos. It has a like a planning system. And what it's just done here is it's thrown up a browser, so we can actually check uh how the application's run. And this is also is controllable by um uh by Antigravity itself. So, if I just do that, if I give it 2 seconds, it should take control over the browser, and it should actually try and run.

So, what it's probably doing now is it's probably looking at the implementation already to see what's actually already done, and it's going to then analyze it. Should we just have a quick look, see what it's actually done there. Uh let's try that again. It'll pick up. So, it says like So, it wants to try and analyze it. Oh, uh one thing that it can do is it can actually inspect the DOM. So, it can look at like the actual web page itself as part of that, and give and feed that back.

And when it finally finishes doing the task, it'll give you like a report at the end as to what it was able to achieve and what it implemented, so you can then review. It can also capture, for instance, like a a screenshot or like a video. So, if it's like an interaction, if you say like add this feature to my web page, it will go through and then actually try and run it. I don't know how good it is at playing games, but we're about to find out.

It's either going to try and play the game or uh Let's have a look. Let's see what it's doing at the moment. It hasn't loaded it yet. Let's go back. Okay, we've got an implementation plan here. So, this is what it thinks it should change about the file. So, what you can do is you can just go in and you can like edit a line and say like, "Actually, I want this different behavior." Or that's not what I meant at all. So, it kind of gives you that human in the loop feedback.

And then when you're done, you just say proceed. And then it will go away and do that. Um while we're waiting for that to happen, do you want to say a little bit about what you work on and like how this kind of relates to the your kind of day-to-day job? Yeah, sure. So, um one of the things I I worked on a few months ago was the the deep research agent, which is now available via the the interactions API. Um and so that's that's been great, but as we continue to iterate on that, my my focus has turned now to making best use of of this anti-gravity harness internally.

And so, of course, that that applies to scaling it for for all of the coding we're doing and, you know, we have a gigantic mono repo, so it's pretty complex. Um but now starting to think about how we generalize this to a variety of other use cases. So, potentially deep research itself rather than passing around huge huge blobs of of text from from the searches that have been done, why not have the different parts of that pipeline collaborate in in a shared file system?

Um and so, yeah, that's that's really been the focus for me, really um tightening up this harness and and making it excellent at at not just coding, but a variety of other tasks, too. And do you have any interesting use cases for like how people within Google or DeepMind are kind of using it at the moment? Like, what kind of what kind of things are they doing with with the agents in DeepMind? Yeah, so so there's a huge amount that's that's been going on.

Um we have quite a few exciting things that you'll probably learn more about in a few months at IO, which I uh can't go into too much detail about right now. Um but at least internally there's been a huge um amount of focus on on building up a huge library of of skills that that enable folks to to do their job better. Um and you know, skills are great, but in an organization as large as as Google, um there's a risk of skills really sprawling out out of control.

Um and so that's a that's a big area of focus for us right now is um improving those skills, making sure that only the best ones uh survive really, almost Darwinian nature. Um but it really is helping folks to to deliver good code at a way faster pace, which which is obviously awesome. Awesome. Thank you. Uh so you can see this little blue bar around the edge of the moment. That's the uh anti-gravity taking control of the of the game.

It seems to figure out how to start the game. I don't think it knows what the controls are, so may might end up looking those up. Um but yeah, this is So this is what it edited. Uh the file that was there before was actually generated by a different model, so I can tell you that it's not even the same game. It's completely rewritten it from scratch um based on the spec we gave it. Um and then what you would get at the end Oh yeah, you can see that it's like actually looking at the DOM, looking for any areas, trying to figure out how to actually use it.

You should get a video at the end here. Uh oh yeah, this is a it's scratch pad. This is like it's notes that it's writing as it kind of goes through and does the task. So you can like you can actually get a bit of a trace as to what behaviors it's trying to figure out. Again, you can go in and you can review these things. If you don't like what they're doing, you can kind of interrupt it. So in terms of like the workflow, this is kind of I guess pretty common to like a lot of different agent harnesses at the moment.

Um but this is how we kind of think about using using the things like Gemini models um, as well so that we can use it for our own development. Um, I will close that off now. So, I guess one big question on my mind related to like uh, agents. How do we do things at kind of Google scale? So, if we think, you know, research deep research is a feature within Gemini app, but then also for everyone within Google to use it like what kind of challenges come along with that kind of scaling of agents?

Yeah, so I mean the the the thing that's kind of top of mind for us at the moment is um, how token hungry this this stuff can be. Um, and so as you can imagine just just managing the quota on a on a per user or or per team basis is is really quite um, important. Um, and so there's a lot of work we're doing around Okay. making that uh, more efficient and and and lower cost. Um, and I think we're Ian and I were chatting before and um, I think what's going to be really interesting for for folks like yourselves is is mixing and matching between models like Gemma 4 which which are effectively free from a quota perspective just using your whatever GPUs or TPUs you have.

Um, and then, you know, using the more advanced uh, models for for specific uh, components of the agentic system. Um, and so also e- e- evaluation as well is is a big um, thing we're focusing on at the moment. Um, particularly with these really complicated uh, workflows, how how do you actually evaluate that it was successful? How do you minimize the cost of that? So, looking into things like mock TPUs so that, you know, you can test the harness itself and and the agentic flow, but not necessarily using up a ton of uh, TPU hours.

Yes, cuz I mean, I'm sure people are aware of this, but like definitely limited in that capacity at the moment and within the world trying to get enough compute to do a lot of this stuff. Uh just quick show hands from the room, who's who at the moment is either building their own agent architecture or harness at the moment? Okay, fantastic. For So, just quick question, like what kind of scale are you looking at? Who trying to Who are your customers?

Anybody? Okay. Got you. Is anybody using any harnesses off the shelf or you kind of built your own ones? Like, what's the general vibe? Oh, nice. Nice. Nice. And how many users do you do you expect to be able to scale that to? Okay. Wow. That's some scale. Yeah. That's the Yeah, the the challenge, too. I'm sure that's the challenge for us, too. How do you stop like one user from Well, actually, okay, I'm going to turn that into a question.

How do you stop one user from taking down a whole system because by spawning multiple instances and multiple cuz I'm sure that's like, you know, the more we get better at doing this these tasks and like we've got power users kind of spinning stuff up and they've got their team of 100 people like working for them, 100 agents. Like how do we kind of manage like the per user like behaviors if you see what I mean? Yeah, I know it's it's it's a great point and and like I said earlier, honestly right now it's kind of brute force with the the quota.

So we have we have some real power users at DeepMind and ultimately it gets to a point where it's like, okay, you've got to just stop right now. Um but you know, in general I think that raises an interesting point about how this stuff is going to be priced in the future as well. Like you know, you saw Anthropic blocking the the open claw stuff. Because these these agentic systems are so token hungry and the subscription model doesn't really work for that.

Um and so yeah, I think that that's really top of mind for me as well right now how to how to mitigate that. Yeah, I I always used to joke when I joined Google that you know, you've got all these resources, you know, available in data centers to use for different projects. Like how do you know when too much is too much? And one of my colleagues once told me he just said, "Oh, they'll tell you." And I'm like, "Who's they?" And yeah, sure enough there's people monitoring these things 24/7 like, you know, looking at spikes and graphs and all our SRE SRE team and and they do just like reach out to you and say, "Can you just like stop this job running on this one cluster, please?" Uh so yeah, I think yeah, that's also an interesting one.

Any questions from the audience? That's a fantastic question. Um so we built a um sort of custom uh web app for that essentially. Um and essentially there's there's one uh agent back end system that is used for for a lot of stuff at at Google and essentially anytime a user issues a a query to to an agent hosted on that system, it then sort of automatically appears in this UI where you can drill down at various levels of of hierarchy each of the pieces of the of the system if needed you can drill all the way down to the sort of raw predict requests made to the model and and so forth.

And so that's been useful. And we also have a concept of like an agent trajectory store as well. That's more focused around the the coding piece where obviously you can have a huge number of of steps going on there. And it can be really important to kind of diagnose at what exact point looping started happening or you know, the model went off the rails. So yeah, it's all custom internally for now, but I'd be interested to hear what what you folks use for for observability as well.

Any other questions? Yeah. Oh, awesome. Yeah, I mean that's a that's a fantastic question. Obviously I can't go into sort of too much detail on on sort of like release plans and so forth, but what I can tell you is yeah, that's something we're actively exploring. Um you know, we hope that it will make it faster, cheaper and and and hopefully better results if we can effectively orchestrate that that deep research using the same harness.

Um and cuz you know, right now without going into too much detail, there's a there's a huge amount of of context that's that's passed all the way through that deep research system, um which gets quite expensive and you know, consumes the context. And so, we're really thinking about, okay, how how do we make each element of this system more like a collaborator as as part of a a workspace, um which is kind of how it would work if humans were were researching something deeply, right?

Um And I think that opens up uh a lot of nice potential for things like um infographics, additional support artifacts, and documents. So, yeah, it's definitely an area of focus for me. That's a great question. I good question. I don't know is the short answer to that. Um the I think we they're not they're a bit more um what's the word? They're a bit more opaque in how they kind of presented. The way The way you kind of think about it is like you have multiple simultaneous ones working on different tracks, but it's not we don't kind of Yeah, the the short answer is it's not as obvious as to which agents are actually working with on a particular task for that.

It's not kind of like a massively parallel system in that sense. It's more like you can give them different um like trains of operation among a particular project, but you can tell them to kind of work on specific things or you can have jobs that kind of overlap a little bit, but it's not kind of Yeah, I don't have a huge amount of detail on the like the specifics of how the sub agents work. I do think that's going to be um the the future though is how do we make agent-to-agent communication uh efficient and then also how do we give us as the human the ability to really shape that and and almost act like a supervisor on a digital assembly line, you know?

Um so, yeah, watch this space, I guess. Question? You're next. Yeah, for me, I I really like uh skills, and and they've been working very, very well for me. Um Uh perhaps this is controversial, but I did always think that MCP may be a little bit of a flash in in the pan. Um I like it for the from the auth perspective. I think that's that's very powerful. Um but for me, a combination of of of skills and uh guardrail CLI um interactions is is is worked really well.

Um and it it speeds up my job so much, you know? I've got a skill for to your point, to debugging raw logs, and you know, it can do most of that from the from the CLI. Um and in a business of our size, the the great thing is we have these skills contributed by folks who are absolute experts in that particular area, and then I kind of I and the agent get that knowledge for free, you know? So, I'm definitely team skills, if that helps.

I mean, we we support both of them, and I think that's the intention going forward. Again, it's like what the community uses. Like, you know, we want to make sure that they work with the harness, work with the models. So, I think yeah, whatever you guys keep using, we'll probably still be supported. It's probably the way to think about it. Yep. Uh should I get the back? Should we go with that? Yeah, I know, that's that's a fantastic uh sort of Um evaluating this stuff is really hard.

Um even just the the mechanical nature of spinning up all of these sandboxed environments, set up in in in the way needed to to evaluate evaluate a particular um problem set. Um I think that the trickiest part is coming up with new data sets, you know? Um there's a lot of good um you know, open source ones that that are good for benchmarking externally, but you're right, for specific skills, the onus is almost on like the the the author of the skill itself to to come up with um some form of testing that, but people are also experimenting with the agents designing that as well.

Um so it's a little bit meta. But um yeah, a lot of work to do in that space. I think we have a question over here. Even in terms of for like the model and the model usage? Yeah, yeah. Yeah, that's a good question. It's funny actually cuz we have we have worse limits than you do because obviously we prioritize customers and not ourselves. So the fact I was like clicking like a hundred times to keep going is because it recognizes I'm a Googler and that's my fault.

Um you know, it's a good question. Um I think there's kind of two parts to this. I think that there will always be limits, especially like within different tiers. I don't have ultra for instance. So, sorry, my corporate account does, but my personal one doesn't. So, I have I have to change my behavior. But, I think we're going to end up in a world, I mean, we have hopefully somewhere around here is Kevin from the anti-gravity team who could probably talk specifically on that.

Um, but I think we're going to be in a pattern whereby it's commonplace for you to run out of credit or run out of capacity on like one model or something and then move to something else, but do it seamlessly under the harness. So, you can give us preference rather than having like, you know, you've hit your limit on tokens for the pro model, so it will automatically put you onto flash or uh you've reached your limit on everything you have in your in your subscription, so you use a local model or but it but to your point like that it will not interrupt the workflow that you're doing or, you know, the notification you get is not is not the completion of the task.

It's like, "Oh, we've run out of quota, sorry." Like, you know, what if you were off doing something else while you'd sent it off doing a job and you come back to find that it spent the last hour not doing anything because you hit a limit. So, I think the way when I spoke to him last about this, it was very much more about like trying to make it clearer when you get to that point and making sure that you can So, yeah.

Oh, we have time for I think one more question. Do you want to have a go there? Yeah, so um you're you're right. Um, one and I really like the fact that we have some really good stuff in place for that already. So, um the way it works is, you know, on a per language basis, we have a specific like auto review model that has been um fine-tuned on all of our style guides and and and all the rest of it um and previous like good examples of of code.

But then also on a PA or or or product uh basis, folks will come up with their own um like specific SRIs and and prompts and so forth to um to make sure that um you know, yeah, the the the other reviewers get a good signal for for how good the code is. Um and just yesterday actually I sent a PR for review and um and and I didn't even have to trigger the auto review thing. I just got an an agent that someone had spun up commenting on uh the PR with uh quite a good suggestion.

So, yeah, um I think it's important I think to the point earlier about us being supervisors of uh digital assembly line. It's like how do we get help with that piece of it as well? Then we can all go sit on the beach. Yeah. Yeah, we to your point like we you can imagine like the scale of all the Google engineers submitting hundreds of thousands lines of code now being done by agents submitting even more code with more review.

They've built a lot of kind of infrastructure for us. But then also we have like tools like Jewels for instance like which are if you've ever played with that it's you've got like a web interface where you can go and do that on your own PRs in GitHub. So, like and you get review components of that. So, yeah, I think this is I think this is an area that's going to be with the ballooning of I think there was a comment yesterday about like uh the amount of the trillions of lines that GitHub is getting at the moment generated by agents and like that process.

So, as much as we hate our own boring work, I'm sure the agents hate their boring work, too. So, we've got to figure out a way to do that. But um yeah. Um I think we're out of time for questions, but we will be around to chat afterwards if you want to come and join us or if you want to head down to the DeepMind booth later, we'll be around there, too. So, thank you very much for listening. Thank you all.