Prompt to Pipeline: Building with Google's Gen Media Stack — Paige & Guillaume, Google DeepMind transcript

My name is Paige. I started doing machine learning a long time ago. Um around 2009 2010. Um was primarily working with um though it feels like forever ago. I was just talking with a friend about this recently. Um, back in 2009 2010, it was kind of wild that companies would even trust open source software to do business critical work. Um, and so I was contributing to things like numpy, sci-fi, like little like little antenna these microphones.

Um, >> yeah. >> Sure. Cool. Cool. Um so numpy, scypi, uh mapplot lib, which is still just as excruciatingly painful to use. Um uh scikitlearn, kind of the early days of the scientific computing stack. Um and eventually started working as an engineer. Um my background is geoysics and applied math for undergrad and then computer science and carbonate geology for grad school. And so I started at Chevron doing work in subsurface geocciences um doing a lot of things like velocity modeling, drilling optimization using very very basic machine learning models um and uh also some large scale compute.

So, if folks in the audience um also have gray hair like me, you might remember Cloudera uh which was one of the first companies that made some of the open- source uh frameworks and tools available um for consumption of like Fortune 100 companies. Um so things like Spark um a lot of the data bricks team came from the Cloudera world. Um and so I was spinning up clusters of machines there. Um eventually TensorFlow got released in the open source world around 2015 towards the end of it.

Um and I started contributing to that. Um the geocciences world is really big into GPUs. They were even into GPUs before um the machine learning world and um so so I had experience working with CUDA and and all of the the associated tools and TensorFlow when it was first released um only supported CPUs. I'm not sure how broadly that's known, but uh but it was like distributed deep learning across CPUs and they needed somebody to help with like getting GPUs um to to work with TensorFlow.

Um that's also why there are three different code paths in the original TensorFlow 1 framework um because they had to like gut the backend and replace it for CPUs, for GPUs, and for TPUs both for single node and for distributed computation across multiple nodes. Um, so that's basically it. Like I owe my entire career to open source software um, and open source models. And I I got hired at Google for specifically that reason.

Um, and then I left to go work at GitHub for about a year on VS Code, which is also open source, um, and early user experience testing for C-pilot, and then came back to work on our large models. Um, and so I was part of the original Palm 2, Gemini, and Gemma teams. Yep. So that is journey. I'm not sure. Like I said, I'm not sure. It's interesting. >> You need to leave soon. You said to have a >> Oh, uh 4 4:15 because I've got meetings uh we have leadership team meetings from like 4:30 to 5:30. >> Yep.

And Gon will be here. And then also potentially uh potentially our colleague Ian will come to uh will come as well. >> Potentially because I say something hours. I think I think there will be there will be like enough good stuff. >> Yeah. Like if you're if anybody's curious like open source is a great way to like work with the team to figure out if you would want to work with them and then also make sure that your public >> I'd love to show you my >> guess how many stars you got. >> Wow.

Excellent. Good job. >> I'm catching up with you guys. right? Like they're um usually if you see everybody sprinting to do the same thing, that's a great indication that it's the wrong thing. Um and so so like as or or like a thing that eventually the model will will have that capability to do. Um so so one of my favorite examples of this is that um you know when the models were first released they had context windows of like 8,000 tokens 16,000 tokens and so everybody was sprinting to build a vector database um because they were like oh well we have to work around this constraint that the models have this very small context window um and then you know obviously that's expanded over time.

Um there's another great uh example of you know everybody sprinting to build fine-tunes of models to support different languages. Um though now models support a variety of languages. If you've been uh kind of listening along for some of the Google DeepMind sessions today, you've seen that in action. And then also many of our competitor models also support multiple languages. And so I think like we we also saw everybody sprinting to build an agent framework when the reality is I think that the m like all of that will probably be absorbed into the model eventually.

Um and also everybody sprinting towards things like building MCP servers whereas now mostly people have kind of moved away from MCP servers and are adopting skills which are just fancy markdown files. Um, and then obviously longer term you could imagine like, oh hey, I just want a little listener um, watching everything that I do and it will automatically create skills behind the scenes for me to use. And we're starting.

Yep. Yep. >> I was going to say I do not necessarily agree because you guys will always cater the most applications but very very specific ones like for instance which is where I'm working Google is not going to work on that. So those kind of >> so so counter counter example um our first implementation of Palm 2 and our first implementation of Gemini we had to have a fine-tune one called MedLM and one called MedPalM to support medical use cases.

Um, now we see that all of the people who had previously needed to rely on those fine-tunes are just using Gemini out of the box with either retrieval or with a custom prompt because all of the data that we used for those fine tunes is just incorporated into Gemini itself. >> But you have the issue of reproducibility. You're not going to have the same results. You're going to have a lot of consequence. >> So so no large language model is deterministic.

Um so so that will be a problem regardless. Um but I but I do agree like there there are um you know I think a lot of the magic is going to come from having like a very opinionated view of use cases um and being able to work directly with customers and solve their problems. Um so with that and that's a perfect segue into uh into the session today. So greetings everyone. Thank you for being brave and for coming back. Um I know that there were a lot of people who were uh unable to to join us earlier this morning um for some of the sessions that we had.

Um so just show of hands how many folks came back this afternoon were not here this morning or uh like were uh cool cool. Excellent. So uh we have a treat for you today. Um for the sessions that we delivered earlier um we're going to do basically a recap of them. So, we'll be walking through all of the all of the examples. Um, you won't have uh you won't uh uh kind of walk away feeling like you've missed anything. Um, and uh I'll be doing some within AI Studio and anti-gravity.

Um, and then my colleague uh my colleague Gom and you can also see one of my agents that's kind of doing its thing with computer use and browsing. Um uh the uh I'm going to disconnect for a second so we don't accidentally see something that perhaps should not be shared. Um and then uh going to go straight into slideshow view. Awesome. So uh greetings everyone. My name is Paige. Um so I I am really excited to be here today.

Um, and I am especially excited to show you all of the things that we've been doing at Google Deep Mind over the course of the last year, um, last, uh, six months. Uh, it's been a wild ride and never in my career have I been more excited to be a machine learning engineer working in this space. Um, just over the last month and a half, uh, it feels a little bit like whiplash. Um, we've released a ton of different things.

Um, so Gemini 3.1 Flash Live, um, which we'll take a look at in a second, uh, which gives you the ability to have a real-time conversation with the model. Um, Gemini 3.1 Pro and Flash Light, um, respectively, our largest and our very, very small but very capable model. Um, that are also very coste effective, very performant. Um, Augment Code, if you've heard of them, they're a company over in the Bay Area. They've just recently replatted their entire agent infrastructure to default to Gemini 3.1 Pro specifically because of performance and cost ratios.

Um it can do a lot of really good work in a very like um small number of pennies. Uh also Nano Banana 2 for image generation, image editing, including reverse image search, which we'll also take a look at in a second. Um and you'll be hearing about from my colleague Guom, who's our expert in generative media. um our embedded 2.0 model which allows you to embed video, audio, images, code, and text in the same embedding space.

Um so you can say like show me everything you have related to llamas and you will see everything from um like stuffed llamas to pictures of llamas to videos of llamas to things of like what does it sound like when a llama makes noises cuz I don't know but like the model the model somehow does. Um, LIA 3 for music generation, Genie3 for world model building, um, our full stack runtime with AI studio which allows you to incorporate database OOTH.

Um, a custom API keys if you want to use other services and a whole bunch of other things. GIMA 4 for open models. Um, we just released it last week under an Apache 2 license which is really exciting if you care about open models. And then also VO3.1 light for video generation. Um so massive number of things across a broad spectrum of surfaces. Um and part of the reason for this is because Gemini is kind of unique in the industry in a couple of ways.

One of which is that it's natively multimodal. So it can understand video and images and audio and text and code and all of the above all at once. Um but it can also output multiple modalities. So it can output text, it can output code just like all of the other competitors on the market. Um, but it can also output images, it can edit images, it can output images and text interleved and it can also output audio tokens.

Um, so uh, so pretty pretty compelling use cases. Um, but I think it's a lot more interesting to see it rather than to just have me waxing poetic about it. Um, and for that I am going to go into AI studio real quick. Um, and we're going to take a look at some of the things that Gemini can do. Um, so first off, how many people have used AI Studio before? Um, excellent. I'm very very glad that all of the Deep Minders have raised their hands.

Um, but, uh, for folks who have not, uh, AI Studio is kind of the best place to go to get access to DeepMind's models as soon as they're released. Um, we have a playground feature. We also have a build feature, which we'll see in a second, which is very similar to vzero.dev or lovable. Um, you can select different models here off to the right. Um so you can see if I click on the model name um we see some pills um everything from Gemini to live capabilities um image uh image uh generation video generation audio generation and music um and then also our Gemma open model family which we'll also take a look at in a second.

Um you can select different models for the purposes of speed I am selecting Gemini 3.1 flashlight preview. I am also on my personal instance of AI Studio and this is an attempt to drive down costs. Um so Gemini uh 3.1 flashlight is around 25 cents per million tokens analyzed. Um which is really really good. Uh almost an order of magnitude lower than our Gemini 3.1 Pro model. Um and it can still analyze video, audio, etc. um uh it's just uh a lot more lightweight, which means that you might not always get the the same great capabilities.

Um but it usually does a pretty good job. Um and you can do everything from analyzing images to analyzing video. So if I as an example um was going to look up a was going to look like a dinosaur YouTube video. Um we already used this Rexy the the little T-Rex before. Um so I know it works. Um but uh but I am going to find um maybe this one uh carnado uh which is >> uh which is it looks like a very very long video so around an hour long.

Um we might uh chop it up a little bit. But if you click this plus sign you can see that you can add different files. So you can either add files from drive everything from text files to um uh to PDFs. Um, you can upload files directly. You can record audio live. Um, add camera footage. Add a link to a YouTube video, which we'll do right now. Um, so I paste in the link to the YouTube video. I'll do a start time of 0 seconds and an end time of around 300.

Um, and it samples at around one frame per second. Um, and then what you see is this is, you know, 30,000 um, uh, 30,900 tokens. I'm going to make sure to turn on Google search grounding off to the right. And I'm going to say, please create a table with timestamps for all of the kinds of dinosaurs that you see in this video. Um, make sure to include a fun fact about each dinosaur. Um, for all of the other dinosaur experts, um, like armchair dinosaur experts in the in the room, you might have noticed that it said Carnonosaurus.

I am skeptical that that's an actual dinosaur. Um, so, uh, we'll see. We'll see what happens. Um, but what's, uh, going on behind the scenes is that the model is getting sent this YouTube video for inference. And it's not just the metadata associated with the YouTube video, it's also frame by frame um the video itself. And a question >> um like uh does anyone else have uh does anyone else have issues with seeing the settings or like uh could we could we change the lighting?

Oh, is this better? >> Oh, excellent. Awesome. Um, so so that's uh thank you for thank you for the request. Like it's I I did I can see it perfectly on my screen. Usually my eyes get a little irritated by light mode, but this is um as long as it's better. And so um so it took in the video. Um it defined the different dinosaurs that it sees. Um it says that the name means mediating bull. Um mentioned triceratops. Um, and then also mentioned that pyranodons are not necessarily dinosaurs.

They're a group of flying reptiles. Um, and then if I wanted to uh if I wanted to get the code that was used to generate this experience um and I wanted to replicate it in my own app, all I have to do is click get code and it automatically configures the model. Um it configures the uh the URL that I have inputed um the kind of the offset for the video the the prompt that I used um and it's in Typescript, Python or whatever your favorite language might be.

So the TLDDR is that if you can get it working in AI Studio, you can get it working as part of your app. All you have to do is click the get code button. Um so that is a Gemini 3.1 flashlight for analyzing video. Um, you can also use it to analyze images with a couple of other bakedin tools. Um, and if we look over to the right, we can see structured outputs, code execution, things like function calling, even for custom functions.

Um, also some things like URL context. Um, and all of these are very very special, but they're also just oneliners if you want to use them as part of your API. Um, so as an example, if I turn on code execution, um, I'm going to see a Gemini 3.1 flashlight selected, um, I'm going to go into compare mode. So I want to compare it, um, with maybe the Gemini 3 flash preview, also with code execution turned on. Um, and I'm going to look for a picture of Lego bricks.

Um, this one. And copy the image. Uh, and right now we're in compare mode. So I'm comparing two different models, both with the same tool turned on. Um, and say something to the effect of draw bounding boxes around all of the green Lego bricks uh using Python. Um, make sure uh or display the image with bounding boxes. And code execution is sort of giving Gemini the ability to uh to sort of stand up a makeshift Python environment that's sandboxed um use a whole bunch of data science libraries that are pre-installed um and use those invoke them as tool calls um kind of like writing the code and incorporating anything that you might share in.

So, I shared in um this image uh and very very quickly, I'm not sure if you saw how quickly um but it was able to draw bounding boxes around the the green Lego bricks, you could also ask for segmentation masks. Um and even more excitingly, so Gemini 3 Flash is still plugging along. Um, but if you look at how much this cost um to do to define the bounding boxes, you could have also asked for things like um tell me how many green Legos there are or tell me uh um you know what are the orientations of the Lego bricks.

Tell me uh how many uh you know what are they called? The little funny rabbit things, laboo. Tell me how many of those you see in all of the frames of this video. um and at what time stamp like those those are the kinds of tools that can be invoked um via the sandbox environment. Um and again like very very inexpensive in order to do this work with Gemini 3.1 flashlight and code execution turned on. Um and it's just a oneliner to stand up that sandbox Python environment um with compute that Gemini can use to do that work.

Cool. And it looks like the uh it looks like the Gemini 3 flash preview was able to do it. It was checking its work. So that was the iteration that it was going through along the way. So it got this first result and it said, "All right, I want to double check and verify that what I did was right. It drew the segmentation mask with uh to like define all of the the like green spectrums um that it saw in the image. Uh and then it kind of verified that those were the correct coordinates.

Um and still if I look um it's significantly more cost with the verification but still on the order of pennies in order to draw the bounding boxes. It just took a little bit more time and also a lot more tool calls to invoke. Um so I strongly strongly recommend um playing around with Gemini 3.1 flashlight for your use cases especially if previously you were relying on Gemini 2.0 flash or 2.5 flash. Cool. Um, so we also have a feature called build.

Um, build again is kind of similar to v0ero.dev or uh or lovable if you've played with that before. Um, uh, we've recently added a feature where you can add database and off to your apps within build. Um, and so as an example today, if I wanted to click this guy, um, and say something to the effect of um, create an app that gives the user the ability to upload an image of their bookshelf. Um, the bookshelf should have a whole bunch of books on it with uh, so we can see their spines.

Um, uh, things like titles and author names. Um, I want you to use Google search grounding to uh to fill in the blanks uh for for the books. Um, so make sure that you have information about the title, um, the author name, the description of the book, and then also what kind of genre uh, it might be. Um, and I want you to save it all to a database. Um, so the user logs in with uh with their Google account. Um, they upload a picture of uh of their bookshelf.

Um, and it saves all of their books uh in like this database format. Um, so they know what books they have and it's attached to their account which is a tall ask. Um, so so it's uh kind of defined that work. I'm going to I've got 3.1 Pro Preview selected. I'm going to do the default instead in the hopes that it might be a little bit faster. Um, and I'm going to click build. And what happens behind the scenes is we get um we get put into this IDE like environment um where you can see the the model kind of going through the thinking process figuring out what it would need to do in order to spec out the assignment um build the plan how long it's working.

Um you can also upload files. So you can upload uh files that you might have like PDFs or specs for apps that you would like to create um and give them to the model as well. you can connect to your drive instance. Um, and we also have a settings section off to the right where you can define custom secrets. So, right now I have a Gemini API key that I've pre uh pre-added. Um, but you could also add like a Superbase API key or um an API key for NADEN or whatever your favorite flavor might be.

Um, you can see a new version history which if you've played with AI Studio before is definitely uh something that is much appreciated. Um, and then also integrations. So things like OOTH, um, as well as GitHub, so you can sync to a public or a private repository. Um, but while this is working, um, I'm going to go ahead and show off, uh, something called, uh, Genie 3. Um, or actually before that, I'm going to show off Gemini Live real quick.

Um, just in case folks haven't seen it. Um, so how many people have heard of Gemini Live? Few hands. Cool, cool, cool. Um, uh, we also happen to have the expert for Gemini Live. Uh, Ian, come on down. Uh, Ian is the, uh, the Gemma the Gemma 4, uh, team member that I had mentioned, uh, that I had mentioned before, and he'll be doing some live model demos, which makes me really excited. Um, but, uh, Gemini live gives you the ability to have a conversation with the model in a variety of languages.

Um, but you can also share video feeds, you can share your screen. Um, and all of this is kind of stacked together in one speechtoext LLM understanding and texttospech pipeline. Um, so as an example, um, uh, we've still got our Lego bricks and pieces pulled up so I can share my screen to say, uh, hey there Gemini. Uh, what do you see on the screen? I see a Google search for Lego bricks and pieces. There are image results showing various kinds of Lego pieces, sets, and different color combinations.

On the right, there's a larger preview of some brightly colored Lego brick illustrations from Freepick. Anything specific you're looking for? Does anybody speak a language other than English? >> Seek. >> See? Oh, excellent. So, so I'm going to I'm going to ask for you to factcheck something. Um, and also spellch checkck something. Um, so only respond to the user in seek. Is that correct? >> I meant Spanish. >> Oh, sorry.

The Oh, gotcha. So, Spanish and Yep. Oh, there we go. Uh, only uh Spanish. Excellent. Um, and then I was about to say I don't know that language or I haven't had that before. Um, but only response >> Spanish. >> I know I know Spanish. Like I grew up in Texas, so it's kind of like prerequisite to know Spanish. Uh but the uh but I'm going to share again. Um hey Gemini, could you tell me what you see on the screen? Or hopefully let me see.

No, I don't think to click the microphone again. >> Oh, it's the There you go. Like the uh So, one of the things one of the things that you can do so you see that I've modified the system instructions. Um but uh like do you speak a specific dialect of Spanish? Castellian Spanish. Okay, >> excellent. So, um I removed the system instructions and I should be able to do this just within the span of conversation. Um so, hey Gemini, could you tell me what you see on the screen, but could you do it in Castellian Spanish?

Excellent. >> Yep. Awesome. So, so you can uh you can ask within the span of the conversation. You can um modify the system instructions to select different languages or different dialects. Um, and then again the same thing if you click get code, um, it gives you the code that you would need to use to replicate whatever you just did. Um, so the model name, any configuration settings, um, as well as any tool calls that you might invoke.

Um, and it works with sharing your screen. Um, you can kind of interact with your screen as you share it. Um, but it also works with video feeds. So you can say um hey Gemini, how many fingers am I holding up? And also compose a poem about me. Well, I see two fingers up like the peace sign. And here's a little poem for you. With golden hair and an open heart, you come to learn, to play your part. The cameras focus, moments start, a creative spirit, a work of art.

How was that? That was very sweet. Thank you. The And so the uh the models are able to to kind of interact uh to view video feeds to to view the the screens. Um and you can stitch them together in your own projects. Um just taking a look. I'm going to enable Firebase real quick. Um so it should be setting up the database for that app that we were building. And then the other thing that I wanted to show is something called Genie.

Um, so if you haven't heard of Genie before, this is a world model that DeepMind has created. Um, it's actually a composition of models. Um, so Nano Banana, VO, a bunch of Gemini used for prompting. Um, and it's stitched together into a system that allows you to describe something um like a a game, an environment with a character um that you can interact with that you can uh play this game uh and and sort of do it pixel by pixel.

So it doesn't generate a Unity environment. It doesn't generate an environment for Unreal Engine. Um it just generates this uh kind of frame by frame experience um for anything that you can imagine. And co. So it could look something like this volcanic landscape where you're navigating with a little rover um with your arrow keys. Um or something like this jet ski where like you hit a light um and it feels like the physics is happening or like the physics is uh like responding in a real way.

Um if you knocked that light into the water and then circled back around, it would persist throughout the duration of your 60-second experiment. Um but there's no physics engine behind the scenes. Um, and then even things like experiencing a hurricane in Florida, you can start with a static image or like a family photo. Um, and see how uh see how some of these things get created. Um, but for this, I am going to go back to my other browser.

Um, I'm going to pull up Project Genie. I'm going to click explore now. Then I'm going to say something like uh a world. Um maybe a Regent's Canal on a sunny um day. Um but with dolphins uh swimming in the canal and uh all of the boats have pirate flags, um which is hopefully not part of the training data. And then uh the the uh sort of character description would be um you know something again kind of atypical maybe a pink sparkly squirrel um uh with um a pink sparkly squirrel with a purple um purple feet and um a pirate hat. uh and then create the sketch.

And what should happen is that it uses Nano Banana to to kind of ideate on that first frame. It'll show it to us to to make sure that it looks consistent with what we had described. Um uh clearly like Regent's Canal is getting overtaken by pink sparkly squirrels with pirate hats. Um, and then what uh what we should see is kind of this first iteration and then a playable world that we can interact with for just 60 seconds, at least for the um at least for the first implementation that we've released to the public.

You're able to access Genie 3 through an ultra subscription in some parts of the world. um not every part um but uh but hopefully hopefully I haven't over booked my uh my GPU or TPU aotment or allocation um if we have uh we can take a look back at the um we can take a look back at shelf scan. It looks like the model is doing the work of creating the creating the fire store rules for us. Oh, there we go. So, pink sparkly squirrel, um, pirate hat, regions canal, dolphins.

That looks pretty good. Um, so let's go ahead and create this world. Um, you can use the arrow keys to move around the, uh, or the WD keys to move around and then the arrow keys to change the perspective. Um, and then we should also be able to use the space bar to jump. Um, but let's see. Let's see how this works. Uh, oh gosh. Whoa. Whoa. Whoa. squirrel. Um, and so it looks like it's walking on water, the squirrel. Um, or like uh hopping along.

Um, you can also jump. Um, so jump on top of the boats. Uh, you can see the the little bicycles. Um, you can see some of the people along the way. Um, and it does look like all of these boats on Regent's Canal um have uh have pirate flags uh and dolphins um that are not currently moving, which is which is pretty wild. Um, and then if I click uh spacebar, you can see the squirrel jump. Um, it looks like it doesn't realize that Regent's Canal has pretty deep water, so I probably should have specified that in my prompt.

Um, and then you can also see it uh attempt to jump. um uh attempt to jump into the uh onto the sidewalk uh and do do its work. Um so it's wild to be able to see the things that you can create. Um the different uh the different experiences that you can construct. Um and again, it's even more bonkers to me um that each part of this is being generated dynamically um as you're moving your arrow keys around. Um so it creates a video at the very end that you can download that you can uh kind of see and review and interact with.

Um and uh this is again just using Gen3 and this composition of models as opposed to a singular model. Um other world model building companies so things like World Labs um that's Fe Lee's company are taking a slightly different approach. They're they're building out actual Unity environments or Unreal Engine environments. Um, none of these things are stored as 3D game assets. Um, they're just kind of like raw pixels um that are incorporated into the into the experience.

Cool. So, going back to AI Studio, it looks like the app is still getting uh the app is still getting cooked. Um, let me uh and hopefully hopefully we'll be able to see it. Usually, whenever it starts working on config files, that means that it's almost done. Um, I also really love looking through it to see um like what its approach towards the the construction of the fire store rules were. Um, what its approach towards prompting the model might be.

Um, it looks like it's confirming the app. Um, and then uh once it's done, it should make a little bitty noise um to uh to talk through um to talk through the the app itself. Um, so it looks like it's ask asking to allow my camera. Um, so I'm going to allow. Um, we have this shelf scan AI experience. I'm going to sign in with Google with my personal account. We can see that it's connected to Fire Store. Um, I'm going to find very quickly a bookshelf um with books on it.

Let's see. Those don't look like real books because many of them are hanging like suspended uh below the shelf. Um like AI AI image generation makes it hard. But this is uh this one looks decent. So like let's save this image. Um looks like somebody has a whole bunch of cooking books. Uh I'm going to upload the photo. So this pixel photo um and hope that it can understand uh hope that it can understand web pages orwp um rather format.

Um it's gathering data via Google search. Um and then the books should be populated in the library hopefully. If not we can try with a JPEG. But it does have pretty good branding. Um, it was able to identify the seven books it looks like. Um, or at least like it identified the books. Let me try uh let me try again but just with a JPEG image. Um, so I'm going to just take a screenshot. So same image just stored as a screenshot.

I'll find in here. So, desktop screenshot at 417. Then identify books. And if not, we can try fixing the errors as well um to see what might have been going wrong. Oh, so so it looks like the uh there are insufficient permissions um for saving in fire store. Um so uh so it looks like it's going ahead and fixing fixing those issues. Um but as it does um you can also see that you can log in, you can log out, you can share the app.

Um so you can specify who has access to it, share full screen. Um and one of the things that I also really really love about AI Studio um is that uh it's figuring out uh where the files should be modified in order to make those changes. Um so it's figuring out the validation logic. It's figuring out that the that the size um might have been the corporate um so the the image URL size um and then it's figuring out where it would need to modify in order to uh in order to make that change.

Um so it looks like it's in the fire store rules. Um some of the other nice things about AI Studios build feature is that we have um an app gallery. So if you need to get inspired for some of the apps uh that are using our models, you can kind of review them. Everything from LIA for music generation to um multiplayer experiences with games. So you can see this multiplayer neon snake or this mandal bulb explorer. Um you can see a design with nano banana.

So like being able to change and modify images. Um, you can take a look at the sik uh media pipe example. Um, which allows you to to kind of play this game where your hand is detected. Um, but you take this little uh and everybody can find out that I play this game really poorly. Um, uh, how to uh, uh, move one of the little one of the little marble deals. Um, oh gosh, I am really horrible at this game. Um, but the uh and then it also sounds like the other app uh finished getting created.

Um, perfect timing. Um, but if I if I sort of upload the image, let's try to identify the book again. Fingers crossed. Yep. And then it automatically populates all of the books. So they all got uh they got all got cataloged with the date, the the type of book, the details associated, um the name of the book, the author, even though some of those were not uh available in the in the spines of the books that I uploaded.

Um, if I log out and then log back in, um, it keeps all of the the books that I had added persisted. Um, and if I wanted to share this with all of you because clearly I want to know what all of y'all have on your bookshelves. Um, I could copy this link and then do a QR code generator. Um, and uh, if you use this QR code, you should be able to access the app that I just created, upload your own bookshelf images, and then have them cataloged um, to your own uh, to your own apps.

Um, next feature would be like finding a way to give my friends the ability to request them. Um because every time I give my friends a book um they they have a tendency to keep it which I understand but is also like exhausting. I have so many copies of Infinite Jest out in the world. Um but the uh but this is the um like uh just kind of a whirlwind tour of some of the things that you can do in AI studio some of the new features that we've added the new models that we have available.

Um and with that I am going to welcome my colleague Guiam um who is going to tell you all about our generative media models. So everything from music generation to image creation, image editing to video generation. Um and it should be a fun time. Um so thank you so much. Thank you for coming. Hello everybody. You will need the up and this is the right one. >> You're going to be awesome. And then do I have where did I leave? >> You can you can show my screen now.

Okay, good. Um, so as as P said, we are it's going to be the same talk as I did this morning, the same workshop. Uh, we have slightly less time. So I'm going to go faster on some things and maybe not run things uh in real time. But you like the the content I'm going to show is on this uh this link. So if you just you can you can just open it and and run it yourself at the same time. Um, so uh I'm going to talk about generative media.

Um generative media is everything about creating um images, videos uh text uh um like spoken text and uh and things like that. I think I would I would fit Jenny into gen media as well. Um so yeah and we have plenty of models like that at deep mind. So let me let me go through all of them. Um so very quickly my name is Guom. Uh I've been at Google for six years now. Two years at Divine doing uh developer advocacy. Uh mainly most on most of the Gemini model until last year and since and this year I'm focusing more on the on the gen media models because that's the funniest model to play with.

Um I've been I've been working in the video game industry before. Um and that's how I joined Google initially. So uh yeah, open your phones. Everything is a media. I already said that. Basically, um Paige talked a bit about our vision of what um world models are. My definition of a world model is something that can uh ingest as many modalities as as it as it can and understand them kind of for five senses and to to also talk or output things in different modalities as well.

So that really has been at the core of the deep mind vision of what uh generative AI should be. Uh the the first gen Gemini model like it was only two years ago but it seems old but it was only a text to text uh to textto text model but actually behind the scenes it was already a multimodel model because and that and you could have been sending it images but it was blocked. They did for testing reasons. I don't I don't remember why but basically they blocked it in the model with with post training and then when it released a few months afterwards we released 1.5.

This one the new the novelty was that it was uh the first multimodel model. Uh sometimes when you were giving it a picture it was it was answering I'm sorry I'm just LLM. I can't do anything with images because some of the that training was still still in there. Um but basically yes we we want to create those models that can understand all of the physics of the world from videos audios and and all of that. Uh and the only reason why we have so many different models is that it's easier to ship one models that only does video and one models that only does images and one models that only text than to have like one models that does everything.

And then it it becames a problem. every time we want to update one of the feature we have to release a whole new model and everybody has to to convert and all. So that's that's just the reason for that. Um everybody knows us for the Gemini models we have lots of other models I will go over like I will skip that for now. Um quick um quick timeline. Uh we release things all the time. Uh I think Paige says it said it earlier on on average we are releasing a new model or new capabilities every five days.

Uh that's just the gen media models. Uh there's all of the other models on top of that. And if you add all of the changes we are doing in AI studio, all of the pricing things like small features here and there we are releasing like two to three new new things every week. Um and um so yeah, it's so it's basically it's it's hard to keep track and like it's even hard for us to keep track of everything we have to offer. So I know that's that for people like uh like you it's even harder because you also have to look at what the the competition is doing.

Um so that's also why we are doing those those um those talks very quickly. uh the the updates on Nano Banana. We released Nano Banana 2 a few months ago um in end of January, the beginning of February. Uh the main thing is that you can output different uh as aspect ratio and u sizes. Um it has search grounding. So that's basically how I do you don't see my screen anymore. Why? Okay, just imagine in your head that's that's what media ads are for.

Um, so the the main thing about Oh, thank you. Nano Banana uh Pro was that it it add search grounding. So you could ask it to search for the internet and that's how I made this image. Like just look for what you can find about my footprint on Google and make an image about me which is kind of scary in a way. Um and but the new things with Neo Banana 2 is that you can do the same thing with image running. So it can you can uh talk about a specific places a specific like bridge like that and it will look for for an image on on the web and then create image based on that.

So that's without running and that's without grounding. So you can see that it's looks a bit more than the uh than the normal building. We also have VO. Very quickly on VO the main novelty in the past uh week is that we released V 3.1 light last week uh which is the cheapest model for generating video we have. So it's only 5 cent per per image. So it's which is way way cheaper than what V3 was a year ago. So the idea that you can use that to prototype, test your prompts and so on.

And then if you want better quality then you can move to the to the better models. Uh and then LA 3 is the the the coolest model this year. Uh it's all music generation model. Uh so you can either generate uh 30 seconds songs or full songs of three minutes uh with the lyrics and all of that. And I will I will show you some demos afterwards. Um and I've as far as I know we are the first one to offer such uh music generation models through API.

So that's really cool for any kind of uh agentic or whatever workflow you you might have. So if you want to be waken up with a song about the latest news every morning, you can do it. Um, and also another one very quickly that I love but nobody knows about is that we have actually another LIA model that is LA real time and this one is basically a live model. So it's creating music uh indefinitely and and you can just prompt it differently.

So it's just going to change what you are uh the kind of music it's generating in real real time like a DJ. So, it's it's pretty fun to play with, but uh some old like people don't know about it. Um as I said, I'm going to because it's it's meant to be a workshop. So, the idea is that you test things yourself. Uh so, you can open this uh this link and that will show you the the content I prepared. Um while you take pictures and all uh just one disclaimer, it's gen media models.

So, they are all paid models. So they like running the notebook actually costs a bit of money. Um the video the the the video generation is going to be the most expensive thing. So you can just keep that. Uh the rest is pretty cheap. So I think you can run the wall the wall notebook for something like one one. So it should be should be fine. But just I prefer to to be uh to be clear with everybody. So let's let's move to it.

So what's the idea of this workshop? So the the idea was to showcase all of the gen media models with this uh this example of we're going to take a book uh and I'm going we are going to take a book from an an open-source library. So uh we are allowed to use it and then we are going to create images to illustrate uh what the characters look like what the and what's happening in each chapter and then we are going to move to the other gen media models.

So creating videos uh about uh about the chapter and then creating music and having Gemini uh tell us about uh what's happening in the in the chapter. So this is just setup. You need to install the SDK. You need an API key. I guess you can you could have guessed that. Uh I'm initializing the client. And something that uh I didn't know about until recently is that you actually have a way when initializing the client to uh implement some retry system.

So which is kind of uh very useful uh nowadays with whenever you use nano banana too because uh especially when the US wakes up it's becoming harder to uh to get to uh to get something out of it. uh and we're working all of yeah we're working on getting more capacity but still uh it's having some kind of retry system al always helps um so import and then we are selecting the models and I usually when when I create content for that are using paid models I usually have some kind of checkbox that uh you can if you if you're opening it's already checked it should be it should not be but I made a mistake yesterday evening uh because I don't want people to run the models and have to pay uh especially the the VO the VO notebook for example example like cost something like $20 to run so I don't want anybody to run it by mistake um just for the sake of the of the demo I'm also limiting the number of characters images and chapters images we are creating just so that it's it's faster to run and all so the book is uh named the wind of the willows from Kenneth Graham and I'm I took it from the project Gutenberg libraries which is an open-source library where you can download open source books.

For some reason it doesn't work uh since I've been in the UK. So there has to be something about this library not being available in the UK. Um but if you run the notebook it should work because it's very likely the server is in in another country. So it will be able to recover the book. Um so so what I'm doing is that I'm just downloading this uh this book here like I like with a with a get uh URL and um I didn't say I didn't talk about that and page neither.

So uh very quickly uh because everybody's always asking that question to us. We have a specialty at Google is that we are always creating multiple projects that are doing the same thing multiple products and we are doing we did the same with messaging apps. We are doing the same with uh with Gemini. So that's can that can be a bit confusing. So just very quickly to uh to clarify things um so if you if you look at the graph like basically we have the consumers apps.

So on the on the on the left it's the apps that are for everybody. You can do plenty of things with them. You can uh you can ask any question to Gemini. you can generate the images and all but as a developer like I guess everybody in the room you can be kind of frustrated because you don't uh you can't really access the parameters select the exact model that is doing you can't it you know that it's it's likely doing cool tool cool cool tool calls but you don't know which one it's doing so that's uh that's nice for the for the broad public but it's uh it's not really for us um on the other end we have Vertex AI which is our enterprise ized uh offer that's the exact opposite.

You have a lot of control. You can decide in which data center your your prompts are going to be run. So especially in Europe, a lot of people are looking to be certain that that the data is not going to to leave Europe. So that's um that's kind of convenient for that. But it comes with uh like great responsibilities as well. So it's kind of hard to set up. So I usually only recommend people to start with Vert.Ex text if they are already using GCP or if they have a team of devops who can do the setup for them and and in the middle ground we have AI studio and the developer APIs that we made for developers but basically the idea that it's as easy as possible to start to play with the model and and do stuff by just creating an API key and then using it um and as part as easy as possible uh Yeah, as part as easy as possible, we have this uh client file upload API that is basically a way to not have to set up buckets to store your your files.

And uh we are going to I'm using it in this example so that we are uploading it to this behind the scenes. It's creating a bucket, but you just the idea is that you don't have to bother with how it works. You just upload the file and then you can use it in your in your Gemini prompts afterwards. Um, I'm also going to use uh structured outputs because I want to be certain about what exactly the model is going to to do because I I'm going to have it uh to to ask it uh to generate a lot of prompts.

So I want to know exactly that uh it's just returning the prompt and not some like uh introduction or um text or something like like the oh yes sure I can do that and then you it's hard to pass. So that's why we are I'm having this uh this structured output and I'm using chat mode which is basically a way to uh u to chain request to uh to the model so that you can uh it's it basically saves the history and resend history so that the models knows what happens before which is quite convenient in this case because we don't want to upload the book all the time.

We just want to the the book to be in the history and in the in the context so the models can ask uh can answer new questions and generate new prompts about the book. And for the images, it's going to be quite the same by by using this system. We will have all the previous images that are going to be in memory. So the model will be better at uh keeping the consistency of the characters and better and the consistency of the style.

So uh I'm just giving it the book. Here's the booku to illustrate with nano banana. That's all. Then I'm defining a style. So I went I I made it so that you can just do nothing and the the model will come up with the style. But then I wanted to try something else earlier. So it's going to be a dark fantasy style with black and white background and colored characters. Um, and basically I also adding some system instructions for for Nano Banana because I uh from my earlier test I think it's better with Nano Banana 2 and Pro.

But with the first Nano Banana um whenever I was asking it images about books that were in the portrait um format, it it tend to to book covers that I wanted. So it it was always adding titles and things like that. So I had to add some system instructions to make sure that I I don't want borders. I don't want titles. I don't want description. I stay family friendly which is likely to not be very aligned with dark fantasy I guess.

Um and uh and no panels as well because I don't want uh I don't want a comic book. I want just an image for for each chapter. And then basically I will use that chat to ask the model to describe each of the main characters. Uh I initially wrote only the adults but because at some points Nano Banana could not generate uh kids images in Europe but that's not the case anymore. So we could remove that part and I I'm getting this this list of characters and a prompt for each of them.

And then I can just go over uh each of the each of the prompts and ask Nano Banana to create an image that to illustrate that um that that prompt. So that's how I initial I initialize the image chart. So it's going to be a separate chart. Um and then I'm sending all of the all of the uh the prompts to create to to be created. So we can see that uh so that's that's small the main character. So, uh, as as requested, the background is black and white, and the character has colored.

Um, not very dark fantasy, but yeah. Um, that's uh the water rat. That's the toad. Uh, the the badger. The Yeah, the badger. And, uh, that's that's it for now. And then I'm going to do the same thing again like uh now you you have the full book in your in your history. So give me prompts for each chapter to illustrate them. So up I get a prompt for each of those uh chapter and I'm creating images for for each of the prompt.

So here we have the the characters having a picnic next to the river. Then another one's on the road and something's happening. Um, and here in the forest, that's that's the first chapter. And if you look closely, you can see that there's there was a problem here because the toad was not uh represented using the uh the character uh the the images that we we created before. Um likely because the prompt was not clear that it was exactly the same character.

Um and so which is why I I have a second way of doing it which is actually cleaner. And if you were to do that to do that at scale, that's that's how I would do it. Um, and in I'm creating another type of structured output which is which is a chapter which is which has a name and the prompt as well, but also list of characters that that are meant to appear in the in the image. And so I'm I'm running I'm asking again the model to to come up with something, but this time is it's giving me the list of the character.

And that way I can um basically here um for each for each image that I'm going to create also give the reference images of what the character should be looking like. So that's it only has that those images in in its context instead of having all of the possible images that we did before. In this case we have something like five five characters. So it's okay. But if you are in a real book with like 40 characters, that that wouldn't be sustainable to uh to expect the models to actually manage all of the all of the context perfectly.

Um and and and honestly, if I was to to do it at scale, I think I would even create more than one image for each character. Maybe one one image from the front, one from the back, one from each side. So that and and then I would pass exactly the one that I that I need in the in this case. So we can see that the image are looks kind of the same but uh this time the the toad is the right one. Uh yeah this one really look like um so now we can move uh to the next step and we can use VO to create like small videos based on those images.

So I'm I'm using the the VO the largest model because it's uh because I don't have to pay honestly honest otherwise I would use the smallest one. Um but basically we are going to do the same thing. We are going to ask the model to generate a video using the view model. Uh the prompt I'm going to use the same prompt as the one that we used to to generate the image. And I'm I'm passing the last generated image as uh as a starting um frame.

And then I want to portrait and I want 720p because it's going to be smaller. Uh and here we go. So I actually haven't checked what the sounds looks like because I >> stand back. You shall not pass. Leave this place little ones. You don't belong here. >> I think it's quite good. Um but sometime it's not not very good because the model doesn't like the the prompt was just about creating the steel image and the and I think and the model doesn't know what's exactly is expected to be happening afterwards.

So a better way of doing it is actually to reuse the chat the chat to ask the model to come up with the um with the uh with some explanations about what's happening after the image and I'm also passing it the image again so that it knows exactly which part of the of the chapter we are talking about so that it can come up with the with the right prompt. Um and so it like it came up with this uh this prompt mold shivers and clutters his scarfs in terror blah blah blah.

Uh water brats bravely draws it glass and step forward to protect him. So up we can see how it goes and some oh like it's like the the like in this case the results are not really as good. uh and uh yeah and every time every every test I've done somehow when I use the same prompt it's it add it's add text and and not when I when when it generates another another prompt but but it should not have any impact because in both cases I'm just giving a prompt and giving an image but I don't know why it's happening like that um um and then we can use the new LIA model that we generated like that we um shipped last week uh to uh to create songs for each model.

So still the same the same trick again. We are going to ask the model to create the the prompts for that. Uh and then I'm going we are going to use generate content with the laya model uh ask it to create uh song uh thanks to the chap the the prompt and that's basically it. So let's see the first uh the first song should be orchestral acoustic folk music peaceful flowing acoustic guitar and and flute duets. Let's see.

I think it fits. Then the second one, the open road is uh jy adventurous rhythmic with fiddle and acoustic bus. And then the last one is suspenseful creepy melody with stacato pizzicato strings. And I think it fits as well. And I I honestly I think the the model is really good because I it's not very often that you can do those kind of demos without actually checking what's what it's going to be because I run the the notebook before, but I didn't check what the what the music were going to be looking like.

And that's uh so far it worked all the time. Um one of the things that I s I forgot to say about all of that is that the way we are training our model internally is that a lot of the training data for the gen media models is actually made using Gemini. So that's the reason why Gemini is quite good at generating those prone for the gen the generative media models because it's already trained on uh on understanding what Gemini is asking for.

So that's uh that's what makes the Gemini very good at at generating those um uh those prompts. And then uh the last the last model that I wanted to show is the texttospech model. So I'm pretty certain that you all have heard of this one because uh everybody loved the uh integration in Notbook LM where you could create a um um a podcast based on on your documents. And that's basically the same model that uh you can create you can ask it to to talk uh to tell read and what the text you you give it and you can have it you can have two characters with different voices and and so on.

That's that's what makes it uh like nice to listen to. But what I wanted to show you is that there are actually tricks to um to have more than two voices actually. Um and uh and the reason for that is uh I think Paige showed that a bit but you can uh when you're using text to speech or live you can actually do a lot in the prompting to make it speak in in different ways with different accents with uh uh and so on and we are going to use that to our advantage to basically have more than more than two voices in in the same um in the same generation.

So what I did because I was lazy I would not do that. it it was I wanted to do that at scale but I basically asked it to extract one of the dialogue from the book and then to rewrite it as as a transcript of of a play and to replace the name of the character by nar narrator if that's just a narrator talking and for all of the others is are just going to be named character but with a specific style for each of them and if the same character comes back it's it's going to reuse the same style style so that it's going like the same characters are going to have to to speak the same way and have the same voices.

Um, and then I'm defining the voices. So the nar narrator is going to use Sulafat voice whatever it is. And then the the all of the others are going to share the fener voice. Um, and then I'm just asking it to uh to uh to generate the thing. And just one trick here with the TTS model, you always have to start with read this or uh tell me this or whatever. If you just send the text, it's it's going to ignore it for some reason.

So, uh you you need to prompt it to read the text all the time. And but that's also where you can add some uh some context and like equivalent of what would be a system instructions like read this in a scary way or the characters are very excited. So that's where you can you can add those details as well. Um and so you can see the the the text that we got is narrator is saying that then the first characters is going to talk very fast-paced and with a British accent and posit accent um and then and so on and we can see that the same uh the same way of speaking is is coming often.

So it's when the two characters uh going back and forth. So let's see how it goes. small, neat ears and thick, silky hair, the two animals stood and regarded each other cautiously. >> "Hello, >> hello. >> Would you like to come over?" >> "Oh, oh, it's all very well to talk." He spoke rather pettishly, being new to a river and riverside life and its ways. The second animal said nothing, but stooped and unfassened a rope and hold on it, then lightly stepped into a little boat which had not been observed.

It was painted blue outside and white within, and was just the size for two animals, and the first animal's whole heart went out to it at once, even though he did not yet fully understand its uses. The rower sculled smartly across and made fast. Then he held up his forpaw as his guest stepped gingerely down. >> Lean on that now. Then step lively. >> To his surprise and rapture, he found himself actually seated in the stern of a real boat.

This has been a wonderful day. Do you know I've never been in a boat before in all my life. >> What? Never been in a You never Well, I What have you been doing then? He was quite prepared to believe it as he lent back in his seat and surveyed the cushions, the oes, the rollocks, and all the fascinating fittings, and felt the boat sway lightly under him. Is it so nice as all that? Nice? It's the only thing. Believe me, my young friend, there is nothing, absolute nothing, half so much worth doing as simply messing about in boats. simply messing messing about in boats messing. >> Look ahead.

It was too late. The boat struck >> basically you couldn't guess that it's actually using the same voice for the two characters. They are so much different. And so that's that's that's that's really actually a really cool way of like uh creating those discussions with multiple characters. So I can use it. And uh the last thing is that uh as we said like the Gemini models are multimodel by by default. So I was I was using a book uh but you can also send an audio book or movie or some like not too long movie but you could you could do the same things with uh with using all of the multimodel input possibilities to uh to have Gemini create things uh for to illustrate other um other kind of modalities as well.

Um how much time do we have? Um I I said I was going to show you the uh realtime model. So this is a very good a very cool example that Oh no stop. I'm going to show you something be before. Uh Paige made some example of cool things that you can build in ICO. So one of the thing I wanted to try was basically to do the same thing as what I do in the in the notebook which is kind of lengthy and quite complicated stuff.

So what I did is I basically copy paste it. Can you build an app that that illustrates a book using gen media models as described in this Python notebook and I just paste all of the content of the notebook and and build it and we'll see how it goes. um it can take quite a long time like earlier it took 15 minutes so we you might not want to wait for that but uh we I can show you space DJ in the meantime. So uh as I said the model generates uh music in real time and this is uh this is a a demo where they made a a star like universe of stars and the the planets are prompts.

So if we go closer to some planets it should start playing music. And then if you slowly somewhere else Yeah. We Yeah. I guess in the middle uh area. See you. Uh I'm really surprised like people are not doing more things with this model because it's it's so funny to go and there's an autopilot so you can just let it move around for for 10 minutes and then uh listen to the music changing in real time. Um that's kind of cool.

Up. Let's see how it goes here. Like, yeah, it's thinking. But let me show you what it what it did before when I ran it. Oh, and I I kept my laptop open for half an hour just so that it would not do that and refresh the page. So, I guess I lost. Um, but that's basically what it did in uh in uh like earlier just uh as you can see just the prompt and nothing more. a very long prompt up uh and uh it took something like like a thousand seconds so like 15 minutes if I'm not wrong um to create the app but that's basically doing exactly the same thing so I can choose the file and upload it and it's going to take some times but basic like trust me it's doing exactly the same thing uh as as what the notebook was doing and that's really like quite impressive that It's doing that uh on the first try right away.

Um and just to finish I one of the thing I'm doing when I'm vibe cutting uh while it's working is that I always have those uh instructions on how to get uh to get it to generate um applets or apps. Uh, and one of the reason for that is that I want the apps to be as easy for me as to review as possible because I want to avoid exactly what happened to page earlier that something was not working and then you you need to figure out what it is and ask the model to fix it.

Um so one of one of my key tricks is that I'm asking the model to create different files for each uh each feature so that uh whenever I need to review something I can quickly check like I ask it to modify this feature it's it's updating something that has nothing to to do with it. So I know from the start that there's something wrong going there and I will be more for in the in my in my reviews and all. And also some instructions that I I don't understand why it's not by default in any vibe coding uh tool which is add logs because when you need to uh when you need to debug you you just know the error message is not enough.

You need to know what's happening before and what's happening like because you know you need to know where where it's happening. So that's uh I really recommend you to uh to to use those kinds of of guidelines whenever you you vodit things and we can see how it goes now. Oh, and something there. I don't think Paige showed it, but if you are if you're willing to pay, you can add an API key here and then you are going to have more uh more kota to use Gemini when when you are vi coding on on AI studio.

Uh I think I will just let you show your demos and maybe I can show you afterwards how it looks uh after. >> Yeah, sure. for the video model when you generated the video. Is it using your models to generate that or is it just >> Yeah, it's just view at the moment. Is there a way to orchestrate all the models and put them together or do you have to just use those generate? >> Oh, if if you want to use the music generated by Lia in the V the video video for example.

Uh no, you don't have a way to do that. It's it's more it's more because of the model limitations that the VO3 generation was not meant to be able to uh to ingest audio files. So uh so that's why it's it's limiting. Uh but I guess like the future is that we want every model to be able to ingest all modalities so that we you can you can do that. Uh same thing I think uh it would make sense even for Laria model to be able to uh ingest audio models that it can so that it can be used as references as well or even if you want to do multi-turn and say okay I love I love your song but the the ending was a bit uh uh not not epic enough so make it make it more epic so that it would maybe just just update the the end of it.

So yeah, it's not possible at the moment, but that's the direction it's it's going. Uh >> performance really specific for the music in Ven. >> No, I I wouldn't try VO for for any kind of music like I the the the training data for for the background music and like is very is likely very light because it's always doing the same kind of non-m music things. Um so uh no but once again all of those models are trained more or less together uh and share some training data.

So I'm sure the next generation of VO is going to be better at that. It's just that the current one it it shipped a year ago and a year ago there was pl like we didn't have music generation models like we talked earlier about text in videos and uh and like a year ago nano nano banana was not even called nano banana it was Gemini 1.5 image. 2.0 image uh generation something uh and it was uh it was not as good at as as text.

So that's uh yeah it's just uh just a question of of generation of models uh there. Oh and I forgot something to uh while it work um if you want to run the to run the notebook uh because you will have to remove it as well. Um here here when I started yeah I created the chat I I added this line that says service tier priority. It's actually something that we shipped last week. Um and that's a way for to indicate when you are prompting the models to if you want uh if it's not that important you don't you you don't care about the latency but but you but you care about price.

So there's uh a service tier that is called flex and basically that says uh the the request can take a few minutes to go through but it's going to be alic. So it's kind of the same thing as uh as using the batch API. And on the other end, because I wanted to be sure that certain that it would go through today, you can you can say that it's a priority request. So it's going to have slightly higher priority and that uh it should not uh it's it's going to be more reliable, but you're going to pay twice the price.

So uh if you're running it by yourself, you might want to remove that line to save a bit. Um and we can see how it goes here. Oh, and see as I said like this model is in high demand. So yeah, that's um I'm I'm sorry about that. It's it's a the model is too uh to uh too too good people everybody wants to use it and we don't have enough capacity. So if you have if you have any spare TPU to uh to share um we can make a deal.

You have a question. >> Do you plan to put like variations of this model open like general but for art. Um so the question like for the video for the question is will will there be any open weight gen media model basically >> um so I don't think so for image and video generation uh to be honest um one one of the reason I see behind that is all of the um not sec like security but like the um when when you when you generate images or videos um we do a lot of checks about what you are asking for and what what is actually being generated afterwards and we are blocking a a couple of things that are not aligned with our our visions and I I feel like whenever you have an open weight model for that it's it's more like open bar and you can you can have it generate whatever you want so that's I think that's that's going to be where our uh our company values are going to be uh to be limiting us um that said for example for music it would make sense to uh like especially if you want real time, it would make sense to be uh to be uh to be on device so that it's it's it's doing it faster.

So that that's something that might come at some point. Um cool. Yeah, I think we'll see after the Gemma demos if uh if it works better. Um yeah, hope it was interesting. >> Thank you. And so uh Ian is uh basically my my counterpart. I work only on gen media model. Ian is working on Gemma models and he's going to like the biggest release of last week was Gemma 4 and he's going to show you how cool the model is. >> Thank you. >> Yes.

So as G mentioned, my name is Ian Valentine. I'm a developer relations engineer working on the gem models and this is an impromptu talk. Uh we obviously because of everything that's happened today uh we've done Paige mentioned that we've we're going to do two sessions just kind of go through a whole load of stuff. We have a number of Gemma talks for the rest of the week as well. Uh Omar is going to do a keynote on Friday I believe which is going to cover a lot of like the what gem is and why you should be interested.

So I'm going to like speedrun that section. Um and we have another talk tomorrow on uh what we've called sovereign escape velocities. So like how you take ownership of AI and run things on your device or on your own cloud or on phones and hopefully some of what I talk about here is going to be like a realization of some of that. So that's a little bit more about like the why. I'm just going to show you the what it is.

That makes sense. So let's get plugged in while we do that. Which one do I need? Sorry for >> this magic one. Yep. soon as I get a video in. >> You have any audio? >> Uh, no. Let's go with no. We'll try and show the ones without audio first of all. Uh, okay. So, as I mentioned, this is super impromptu, so we're going to go with whatever we get. Uh, but I will go through I'll just show you a couple bits when we get the screen up.

Oh, do you need me to move it to one side? Or can I do a mirror? >> Let's do mirror. >> Let's do mirror. Uh, where? Yeah, you go for it. Perfect. Yeah. So, what I'm hoping to show you a little bit is uh you know some of the things that uh some of the devices that Gemma models can run on. Like I'll talk very briefly about the different size of Gemma model uh what we released in Gemma for last week because it's brand spanking new.

Um and hopefully show you give you kind of a feel of like the kind of capabilities that you can now do either locally or that you can run on like a single GPU that maybe you couldn't like six months ago. There you go. Here we go. Perfect. Tada. So, Gemma 4. Hooray. Uh, so this was released last uh Thursday. Um, it's a family of four models. So, we have what we call the effective models, the E2B and the E4B models. And these are models designed to run like on mobile phones, Raspberry Pies, Jets, and Nanos, kind of like very small low-end hardware.

Uh, the E part of it is the question we always get asked. The effective is uh because the model architecture has a per layer embedded structure which means that the embeddings actually don't need to be loaded as part of the model. So you can have them running on flash um and then you can page in uh the embeddings as they're needed for the model. So the actual what we kind of describe as like the brain of the model is about a two billion parameter about four billion parameter but if you put all of it in RAM it's a bit bigger than that.

It's kind of more like a five and an 8 billion parameter model. Um so that's why it's called effective 2B. Uh we also have a 26B which is a mixture of experts model with 4 billion activated and we have a 31 billion parameter dense model which is our like our flagship kind of big model. Um and both of the two bigger models the 26 and the 31 are designed to be able to run on like laptops and desktops or like single instance GPU clouds depending on like the precision and quantization you need.

So, not going to get into that, but just imagine that a lot of the capabilities you can do right now, you could do on like a MacBook with, you know, enough RAM or a, you know, a 5090 or something like that. That's kind of where we're sitting right now. Um, why are these kind of exciting and interesting to us? Because with these models, we're focusing kind of on the agentic side. So, they now have thinking built in. They are multimodal, so they can understand image, audio, video.

Uh audio by the way is just for the two smaller models. The effective models that run on the phone can understand audio but the rest can understand image and video. And we're seeing uh performances offer models that are in the range of like 10x bigger than it in terms of parameter size. So what might have required a cluster now you could do same kind of capabilities for uh for a single GPU. So that's kind of really where we're seeing um some innovation here.

Um, and what I'm going to jump straight into is I want to show you the demo section. Oh, I I'll show you this one slide. I I like this one. This is quite kind of explains a little bit where where we've come. So, the Gemma 2 models, you can see like right in the middle, we're kind of, you know, particularly good at creative writing, but uh uh not so kind of evenly spread amongst amongst other different capabilities. And as we've kind of gone through Gemma 3 and then now through Gemma 4, we've kind of evened out the overall capabilities of the model.

We've got things that are much better at coding, much better at function calling, action taking is all kind of built into the design of the architecture of the model rather than relying on uh like strong um instruction following capabilities which we find with bigger models is is kind of more important. So the models are designed with that from the ground up. So uh let's go over to uh this is all spoilers from Omar's talk.

So I'm going to go straight to the demo section here. I'll show you this one. So, this is the uh Google AI Edge Gallery app. Uh has anybody tried this app yet? Uh have uh one, two, okay, like a handful of people. So, you can download this on Play Store, App Store. Um it works on Android and iOS. And uh what it is is it's uh it's a way that you can test and try the models. So, we released a new feature called agent skills.

And what agent skills allows you to do is it allows you to set up skills for the models that run physically on the phone. So this model, this is the E2B model is running literally on uh a pixel I believe in this case. And you can prompt it and you can test out the different capabilities. So it's effectively given a set of um uh skills that define things like Android intents where you can actually trigger and call other apps or you can write your own JavaScript skills or like run like a web view and uh you can instruct the model and it will make a decision about how to actually trigger these.

So you can get it to for instance add things to this uh research tracker or you can ask it questions about the research tracker and it will call the correct function to pull that data back. So, we've gone from like a world of just being able to chat with it on your phone to now that you can give it some more ambiguous input and it's able to make a decision about uh what functions to call. Uh the next one if it plays.

Yeah. So, in this uh in this example, we've got a number of different I'll show you that one last. There we go. Uh a number of different skills that you can try out. So everything from just like you know loading locations in maps to uh like uh uh using APIs and other services. Um so you can build these things yourself or you can just use some of the pre-loaded ones to try and understand what it's kind of capable of. And this is like our playground for kind of doing so.

And then lastly I wanted to show that uh this is the 2B model on the left hand side. So you can you could also try things like vibe coding on device. The model's actually quite capable of doing this. You're not going to build like you know a big architectural system but it can do you know small like web apps that understands lots of different languages. Uh you can write things in Python, HTML, TypeScript. Uh it can validate some of the stuff and uh you know you could build like little apps that can run on the on the phone itself.

So you kind of got the whole feedback loop just on in the palm of your hand basically. So this one they just generated like a really simple calculator and importantly it does actually support divide by zero and gives an error correctly. So it's even able to reason about those kind of things and to uh to kind of add I think this at the moment it's got with thinking turned off but if you add thinking turned on we see that you get a bump in terms of the quality of the output in term uh in terms of what the model's actually able to reason about because it adds that planning step before it actually does the execution.

Okay. So, next I'm going to show you a demo running on the device. Um, who here has used or uses LM Studio or uh Olama or any of those kind of tools? Okay, so like maybe like three or four people. Um, so actually this is useful. So, one thing that's really important to us when we build these models is making it compatible with a lot of the tools in the ecosystem. So we partner with uh folks like Alama, LM Studio, VLM, SG lang to make sure our models kind of like work really well with them and that they can be deployed efficiently.

Um LM Studio is kind of like uh a tool for running local model instance. So in this case, I'm just got the 26B model running on my device and you can uh you can load it from here and at the moment this one is configured. This one will take about uh at the moment it says seven 18 gigabytes but if you add for instance the uh enough memory to do the kind of context you're looking at be like 22 GB uh of uh RAM required to do it.

This is an M4 Mac so obviously has unified memory so you can you can run that but you would need if you wanted like full performance you'd need to have a GPU with enough RAM to be able to run the whole model. And as I mentioned, the 26B model is a mixture of experts. So it uh actually only needs uh 4 billion activated parameters. So it's much quicker than the 31B model, but it is more intelligent than just a 4 billion parameter model would be.

So uh one cool thing about this is that with these models, you can serve them uh on your local machine uh via a compatible endpoint. So uh like an OpenAI compatible endpoint or an anthropic compatible endpoint. and then you can use them with other apps. So what I'm going to do here is I'm just going to serve this model uh on port 1 2 3 4 and then I can use other apps to call uh the chat completions API directly on this one.

So the code to do this would be like open AI uh open chat completions and then point it at this server and it would be the equivalent of just using your local machine rather than using a cloud machine. So I'm going to run a uh terminal here. I'm gonna do demo SVG space. So, what I'm going to do is I'm going to create uh one orchestrator instance and then I'm going to create 10 sub aents and they're going to generate SVGs for me.

So, all they're going to be told to do uh actually let me make sure I turn thinking off to make sure it's super fast. Uh glad I remembered that. Right. So, each of these terminals here is a separate agent. And you can see the one the orchestrator is this one right here. Uh so it's farming out those decisions to all the different sub aents and each of those sub aents has begin been given a different thing to draw. So you can see this will be like a very good way for you to do some kind of quick prototyping on your local machine.

Uh you don't need the internet to do this. And the throughput in the corner up there is how much combined token generation is being done by each individual um sub aent. So when they finally put it all together, the orchestrator will just compile this into a page and you get like a lovely array of different SVGs that they've generated. So you can imagine this being any kind of aentic task that you want to be able to do on your machine.

You know, sorting out files, implementing bits of code, doing like subdividing research, analyzing data, that kind of stuff like that. Um, and you can give them all very different jobs to do this. So some of them have finished already. That's good. So something hopefully will pop up in a second. There you go. So there are my SVGs. So Oh, they're not too bad, are they? And as I mentioned, oh, this one's animated. A spinning planet.

So it's somehow fig figured out how to make that look as though it's u animated, which is quite cool. Yeah. And this is the thinking turned off. So if I had thinking turned on, it does a little bit more thought in the planning stage, and you generally get better SVGs out. Like more time spent thinking about it results in better SVGs. Um, so the next thing I wanted to show you is, uh, I'm going to jump over here to open code.

So, as I mentioned, uh, you can use any application that can use an open AI compatible interface. So, if you've got like a programming environment, uh, you can just basically point it at that and you can try the model out to just see how well it performs. So, in this particular example, the all you do to configure this in open code is you just I'm going to kill all my terminals here. by goodbye. Let's get rid of these.

If you go to there is a single file. If I go vim.config open code config.json and I scroll down to where is it? This section here. This is all you need to specify. So you give the name of the provider, what the schema is, the models that you want to expose, uh, and any additional parameters that you need for the endpoint, and then you just literally point it at a URL, which in this case is just my local machine. And that's all you need to do.

And I have another example. I'm not going to run this right now, but we have a guide for how you deploy Gemma for on say something like Cloudr Run or like a cloud provider. So if you don't have enough RAM to run the the quantization that you want, you can just throw it up to a single GPU and you get this this single command here, this one right here, the beta cloud deploy, and this will put the model on a an RTX Pro 6000.

So you can just if you just want to test it out, that's one way that you can test whatever model that you fine-tuned or run yourself. And we have another way to access it as well. If you go to ai.dev, dev that uh Paige and Gim have been showing you. We've also got the Gemma models are available here. So if you just want to prompt them, we've got the two big models and you can test them out there and they support like you know video and file uploads, grounding with search is all kind of built into that too.

So you can test to see how they perform in isolation as well. So really there's no reason whether you have the hardware or don't have the hardware uh for you to be able to give it a shot. Uh yeah, what I wanted to show you was that I had uh generated a spec for uh a little game. Uh where is it? Okay, so I made a game called Nebula Drift. So what I did is I used the 31 billion parameter model to give me just like a spec for a game and I'm just going to give it to my local model to go and implement.

So if I go implement spec uh and then I just give it the file this one and uh sure it's the right model and let's make sure we turn thinking back on so that it's going to actually reason about it too. Uh there we go. Uh and we'll run that. And so why is this interesting? Because we haven't changed anything about open code at all. We've just literally given it the model to try on the local machine and in the open code spec it's going to give it all the different tools that it needs to be able to run open code and to behave in that environment.

So it should think about uh the task it needs to do and then it needs to look at the file system to either read files from the file system or write files to the file system and if it goes wrong I can similar like to any coding harness I can just prompt it and say oh did you check this or can I feed the error back into you and have it kind of reason through it. So this is just it following these instructions. So yeah it's decided to make a directory.

So it's going to use the shell directly directory from that. So, if you've used any coding tools, this shouldn't be too uh unfamiliar, but maybe you've used it with another model. Maybe you haven't used it with a local model or an open model um like running on your hardware. So, I just want to show you that this is kind of possible. Oh, yeah. There you go. It's already created an index file. That was quick. Um it's going to create, if it reads the spec correctly, it should generate two more files.

It should generate a JavaScript one and a CSS one. Uh and then what we'll do is we'll just try to see what game it managed to make from the spec. Um, and to kind of give you an idea, it should be like a an infinite racing game where you've got to avoid some asteroids. So, if it finishes off the code, um, there we go. Should we give that a shot? That looks like it's actually done something correctly. So, if I go run that, where did it write it?

What was it called? Nebula. Oh, Nebula drift with an underscore. decided. Oh, I think it's even trying to change the file path as I look at it right here. It's trying to make edits to it. Stop. Stop editing. Um, so let's just try and run that. Okay, we have a game. Uh, oh, it doesn't run. Okay, so this is interesting. So, what we can do is if we just go uh let's try and run that again. Ah, so there's a syntax problem.

So if I say if I just literally copy this. So imagine in most harnesses you'd have a feedback loop where you could feed it back straight into the coding. But I'm just going to do it manually and see if it can spot its own error. I uh stop. I found an error. Let's check that. Did you actually read that or not? Let's try again. Okay, so now it should now it should uh do some investigation. So probably reread the files, see if it can spot what the typo is or what the problem is and then it would should try and edit it or fix it.

So actually in open code they give you two tools. One which can edit like individual lines and one that can just like rewrite the file. So it it can even make a decision about which one of those it needs to use to do it. Uh so hopefully it's going to edit it correctly. Um and then write that back out. So it's updated the game jet.js and it's editing uh what are you editing something else? Oh um it can also kick off sub aents too.

So if you give it like a specific skills or task to do, it can basically uh run sub aents. Uh the 31B is pretty good at that. The 26B depends like on the task. Sometimes it understands like what sub aents to call and sometimes you have to give it a bit more prompting. But similarly with an environment like this, you can expose MCP servers, you can create skills, you can do all sorts of stuff to kind of like help guide the model.

Um, should we see whether that's fixed anything? There we go. I don't even see. Did that refresh? Nope, not yet. It's trying to edit the game file. Must match exactly. So, okay. So, it's trying to edit a part of the file. I could also just say like, uh, null. Okay. I think it probably got unloaded. Let's put that back in. Let's load that back up. Try again, Gemma. Right. So, if I tell it just output the full file again for game.js.

Uh, where did it put it? It was in that one. So you can tell it like specifically which file it should be reading and writing to. So hopefully it should listen to what I say and then uh regenerate that file. While we're waiting, does anybody have any other ideas for a game to to make? Paige is obsessed with like pink squirrels, but like so you could try something in that vein. Any any ideas for a game? >> A game where you can build your own game. >> Whoa.

Let's try it. So, I'm gonna use a game. >> Yeah. >> Right. Let's go ask. We'll ask the big model. So, if I go uh Okay. Write a spec for a game where you can build your own game inside the game and use this as a reference. So, we're going to build a spec first. Uh let's just go here and we'll use Yeah, we'll use we'll use the spec for the other one as a reference. Let's try and run that while we see what's going on with our open code.

This looks like it's editing the file right now. Yeah, you can see it's processing the tokens right there. If I go to this one, OmniForge. So, what it's what I've done here is I've told it to follow a particular spec. Uh, so it keeps the same pattern. So, like for instance, if I had like a format that I needed it to do in, it's pretty good at following those instructions and kind of keeping to that uh that record. This may not be impressive if you've used any model in 2026 other than to know that it's something that you can actually run at home.

That's for me is the interesting part of this like the you know all the top models can do this. Uh but yeah up until recently not not that you could run yourself. So uh let's leave that doing its thing there. Has it fixed my file yet? It's made more edits. We can go and see what it changed. So it's starting Oh, it added some more delta time. Oh, it's changing the input. Okay. Okay. Well, you can see what it's generating here.

Like it's got like these little player cases and all sorts of stuff. Should we see if it runs again? Uh, while we're waiting for that, right? Are you going to run yet? Ah, you what I could do is I could just say to it, um, uh, let's say, pressing the button doesn't start the game. Regenerate the files. And hopefully it's just going to write them back out. Let's do that. Okay, thinking about it. See how our spec is getting on.

So, let's just get 31B. Can you just implement this spec uh but as a single index.html with the CSS and JavaScript included. See that it can do that. Okay. Uh, and there's one other thing I want to show you. We'll do that in parallel. AI.dev. One thing that the model is actually quite good at doing being multimodal is to understand um the context of the input that it's given. So I could for instance take uh a screenshot of a website like uh if I go to Gemma for deep mind.

Let's go to this one. And if I just go like uh I just grab this and I could just go to the model uh implement this web page web page as a single index.html. If I just upload that file that one. There you go. and try and run this. Yeah. How's our open code getting on? So, it's written the game file hopefully with no mistakes. Written the style file and hopefully written the index file. So, let's give that another whirl.

Right. There we go. So, there's our game except I don't see any asteroids. Okay, we've got a starfield and we've got a movable ship. So, again, this is just what the model has come up with. You could just iterate on this. You could kind of add to it. Uh I'm not going to I could ask it for more asteroids, but I think we're done with this one. Uh let's see how the other ones are getting on. So, this is the this is the game about making a game.

I'm really curious just to see how what it does for this because that's going to be pretty nuts. And then this one, this is generating the Oh, yeah. This is generating the recreated web page. So, uh this is all free to to play around with. Um so, just be aware there's like usage limits in terms of like like shared ownership. Um but you can just use it like to test stuff out pretty easy. Uh that's why it's a little bit slower than the um the Gemini models because it's the uh there's only one poor Gemma model serving everybody.

Um let's see. Is this one done yet? Still going. I don't know. Maybe it's made like a really epic adventure. I think it might just be this might be, you know, ship it as soon as this is done. Uh what else I want to show you? Oh, okay. While we're doing that, I want to show you this cool demo. So, one of our team uh is really into their robotics. So, they uh they made a version if you've ever used um any of the robot simulation tools.

So, this is called Open Duck uh which is just like a little simulator of duck and you can ask it questions or you can talk to it via a uh the E2B model which is running in the browser. So, it's you don't you actually download the model and it runs in web GPU straight in the browser page. So when it tells it to do these actions, Gemma is interpreting like what it's asked the model to do. It looks at what the the robot can do and instructs it to, if you look really carefully, you can see where it says perform action.

Yeah. Before it actually does the action, it tells you what command it's trying to trigger. So you can imagine this actually running embedded on the device or in this case is in the simulator to just like kind of like prove the point. Um, so that's quite a cool one. The other one uh I want to show you was uh the Android Studio team have integrated it into making Android apps. So you've got like a little chat window at the side and you can uh build the apps from there.

So they again they have like a little aentic builder setup. Uh so if you were just building phone apps, you can um use the model. I think they're using the 26B again. Yep, they're using the 26B for that one. And then uh we also have it working with um the uh ADK. So our agent development kit again it exposes like different functionality. It's got a thinking loop. It's got like a like a feedback system. So you can give it more longer running task and it can make decisions about like pulling information and stuff like that.

So all these different environments that you can kind of run it in. Should we see how our find? Okay, this one is done. So, this is going to be this will be our web page. Oh, that's not bad. That looks pretty good. I don't imagine these buttons go anywhere, but for uh what's new. Oh, they I thought I literally thought it had made a video. That would have been a bit in that would have been a bit intense. But yeah, but you can see even just from like the layout, it's pretty close.

It's kind of matched the font correctly. The understanding of the actual page is pretty good. And this is just kind of oneshotted. So, point it at random web pages and see what it can build. And let's see whether this I'm I am now really interested in this game. Let's see whether this actually works. Oh my gosh. Right. Is this Oh, I've got like a full editor. So, I can build like a little Oh, I like that. So, a goal.

Can I run the game? Uh, I can put things in. Oh my gosh. Please tell me that. Oh, yes. The triangle moves. Oh, I love it. Why? Okay, there you go. That's that's a good point to end on, I think. So, yeah, uh Gemma models, uh you know, they run on your phone, they run on your laptops, they run on GPUs. Uh go try them out, try the AI gallery if you want to explore. Uh I'll be around today and tomorrow if you have any questions about how uh but yeah, uh really excited to share that with you and uh enjoy the rest of the conference. because