Agents Without Code: Skills, YAML, and Filesystems Replaced Python — Philipp Schmid, Google DeepMind transcript

Hi everyone. Uh thank you for coming. I know it's the fourth day, last session before the keynote starts again and we are going to do something fun. Uh we're going to look into how files are basically replacing Python. And before we begin, I would like to start with my favorite definition of what is an agent from Simon. An LLM agent runs tools uh in a loop until it achieves a goal. And what we are going to do is we are going to build uh the same agent, the same GitHub PR review agent in three different ways.

And we are going to delete code on the way. Each new version, less code, more files basically. Um before we begin, I would like to quickly introduce you to the interactions API, which is our new Gemini API. It's a unified interface for um running models and agents. So you can use the interactions API to call the Gemini models directly or to call our new agents, which also comes with sandbox. It supports serverside state management, background execution.

So it's perfectly suited for all that's coming in the next years. and um the capabilities it's the same API for tool call multimodality understanding multimodality generation so you always have the same interface might look very familiar if you're using other LLM applications we really try to build something for developers which you like to use uh to build and that's something we are going to do so something little bit different in the interactions API to other LLM applications or APIs is that we moved away from this term based based uh conversation history to steps.

So until I would say a few months ago, most of the applications were really turnbased. Normally you had a user in input and then a model output, a user input, a model output, which definitely works for normal chat application. But as soon as you start to build agents, use reasoning model. We have more than just a user role and a model role, right? So we have like different inputs, we have different types, we have reasoning.

So we decided to like make a cut, make a change and build something really for agents and that's what you see on the flat steps timeline on the right where you have a user input then you have reasoning you have a function call you have a function result and you no longer need to like abuse the user role for passing back data from an environment. So roughly a year one and a half years ago writing agents mostly meant writing a loop in Python.

You needed to define a JSON schema. You needed to define Python functions. You needed to look at the output from the LLM. Need to check if it was a function call or if it was a text response. And then needed to match it against um the type and then like call the tool look of if you get an error and then like go back and forth and let's look at some some code example on how this would look and also run it and hope that uh the demo gods are great to us.

So I built or I let Gemini build a basic implementation of this Python loop. So we have our uh class. We have a run function which uses the interactions API. We have all of the weird complex passing with function calling with uh appending the errors checking if we get an error and then we have uh the result again. And what we need of course for an agent is we also need a system instruction. So there's a separate file for the system instruction.

Very basic. QR GitHub PR reviewer and then of course we need tools and for tools we needed to write those um JSON schemas specifications of description exactly define which uh actions the agent can take and then of course we need the implementation in this case using the the basic uh GitHub API just sending some some requests. So we can run this um in and basically the main main implementation is a very simple uh input interface and we can say something like hello and yes we get back hey I'm an agent and then we yes ask it to review a pull request on the Gemini skills repository and what we should see is like the agent should hopefully start soon sending function calls function results function calls function results but it's very limited to yes uh great it works very limited to the tools we define so if we ask the agent to do something which it does not have the capabilities to it just says hey I cannot do this um which is unfortunate but that's how we were building agents um raw Python code a lot of files a lot of things which can go wrong a lot of code to manage so what happened afterwards um or what we we need to do we have like a token generation We have the native function calling and we must execute the loop.

We must handle the tool routing. We must create a JSON schemas. We must write the Python code. We need to execute the Python code. We need to manage the state. So there's a lot of things we need to do to get an agent running. And then we got agent frameworks. There were many different agent frameworks which abstracted away some of that complexity. One example here is the ADK framework um where you have an agent class now which handles all of the tool loops, the function calling, the retries, the error handling and it made it a little bit easier.

We basically removed all of the boiler plate code which we always needed to write for agents put it into a framework and help people build with it. So back to the demo and um same example. So we go into the CR2 and what is very interesting if you let me open both. So we still have our we don't have our agent file anymore. So the agent went away. We still have our prompt same system prompt. We still have our tools in this case also no JSON definitions anymore because those agent frameworks now use the uh signature of our functions to create those JSON schemas on the fly to provide the model.

So let's stop our um agent. Now let's run our second agent. Similar interface, similar prompt and we should see a similar expected behavior where we have function calls. We try to get the PR data. We try to get the diff, we try to get all of the code we need and it works and we wait for for the agent to yes continue. But similar difficulty here. If I ask it like what's the weather in San Francisco um we should get back hopefully a result like hey I cannot do this I don't have access to the weather API which obviously makes sense because we did not define any tool still very unfortunate because we need to be very explicit on what our agent can do and we all know nowadays that we just want to prompt something and we wanted the agent to do whatever it takes to to achieve that goal.

So what is left for us to do? What does the framework solve? The framework solves the turn taking loops, the routing, the execution mapping, the JSON schema creation for like the different function calls, but we still own the Python plumping. So we still need to write those tools with Python code. We still need to add specific rules or requirements to like make sure whatever we want the agent to do and we need to provide the environment where all of the tools are running, where we want to host it.

So what comes afterwards? Afterwards hopefully comes remote agents and at Google IO we launched the anti-gravity remote agent on the Gemini API. The anti-gravity agent uh is powered by the same agent harness which powers the anti-gravity IDE. Here the same harness very important does not mean the same agent because the anti-gravity agent is a coding agent at the moment and the uh agent available in the Gemini API is a general purpose agent.

So there might be different system instruction, there might be slightly different tools because the Gemini API already has a Google search tool. So we use that what we have built and but very importantly it comes with this new environment parameter and this environment parameter here allows the agent to get access to a hosted isolated cloud sandbox where it can run tools, where it can run bash commands and where it can save files.

And those environments can be um configured. So you can provide sources and sources can be a GitHub repository, it can be a GCS bucket, it can be inline files and of course very important we want to make sure that those agents are secured and cannot use our credentials in any way possible. So we created a network proxy around the um agent sandbox which basically injects the credentials when the agent makes a request from inside the sandbox to outside the sandbox.

So the agent never really sees your credential. It just knows hey I can call the GitHub API and then on the fly we make sure that it received the correct token which you define and you can also limit which domains the agent has access to. So if you want to restrict the agent completely on which network access it can or which website it can access you just leave it blank. By default the agent can access all because I mean it's a hassle if you first need to define where to go.

So we tried to stay simple and of course making an API call is nice but we thought hey people want to reuse their configuration want to reuse their agents. So we added the agents API where you can define your own custom ID you the same system instruction the same base agent the same base environment and then you can create that agent and then you can use that agent in the same exact way as you use Gemini models or as you use the anti-gravity agent by providing the ID.

So all of the existing code can be reused with your own custom agent, with your own custom tools, with your own custom uh credentials, environments, whatever you need for it to to run. So let's look at how this will look for SS code and as a demo. And okay, now 03. And what might be very obvious is that we no longer have a source directory. So the code went away. We have now an agents M uh folder with an agents MD file with system instructions.

So very similar system instruction. The only difference here is that we tell the agent, hey, you have access to the GitHub CLI. So we no longer create specific tools for reading files from a GitHub pull request, for accessing a GitHub pull request. We just tell the agent, hey, you have a GitHub CLI, you have a bash tool, you have file systems. try to use it whenever you think it's important. And since we don't have the CLI installed, we have a very basic bash script in this case which checks, hey, if the GitHub CLI is installed, please use it.

If not, download it and install it on the first turn. So, we go into our terminal and we run our agent here. In this case, maybe important I use a stream version because otherwise we would wait like a few seconds and we not get back any we would not get back any anything back. So same prompt and we should soon see um our function calls and function results coming in. Yes. So in this case since we run inside a sandbox the agent first like explores the sandbox to really make sure hey do we have this GitHub CLI installed and then tries to run it.

It did not find it on the first turn. So it installs it and then we can see the agent doing its work. And in this case it's not using the predefined function calls. It's using the GitHub CLI and it's already existing knowledge about how it works. I have a bash tool. I have like access to the file system and I do all of that work to see or to like review the the pull request. Let's wait a little bit. Okay. And I think the the amazing part here is like if we ask the same question as before, what's the weather in San Francisco?

We should hopefully see that the agent tries to use ah it uses Google search in this case on 2nd of July. Let me quickly check. Yeah, that's today. And we have around 20° Celsius and it works. So the agent became more of a general purpose agent and we don't need to like specify all of the tools. We basically trust the model on understanding hey I have a specific set of very atomic general purpose tools to solve my task or the task for the user.

And if we look at the the code uh for like the the input or like the the sorry the the interface we have our sources here. So we have the the bash script which install the GitHub CLI. We have the agents MD file and then we say hey you can use the GitHub API with credentials. So I want to access or use GitHub credentials in a secure way. So I created a token for the API and also for github.com since you need both URLs. one uses is used for the git uh commands.

The other one is used for HTT commands and then domain all is basically hey in addition to the GitHub URLs you can use all of the web but you don't have credentials for it and then it's a it's a simple single API call to the anti-gravity agent with your or user input with the environment and then also with the previous interaction ID that we keep the multi-turn going and that that's all it takes and it's a single API call on the backend side we start that cloud sandbox we load the agents MD file and the skills from the environment provided to the model and then the model between the API and the sandbox does all of the the looping calling the function returning the function results calling the function returning the function results and that is all it takes.

So where does it leave us? We no longer need to execute loops. We no longer need to do two routing. We have a serverside conversation and session state. So we only need to provide new inputs. The context window and the compaction is also automatically managed by the agent. So if we continue our conversation at a certain point the context is compacted and we can continue without the need to manage anything and we also get an isolated remote Linux sandbox which we can use to run our code.

So what is still left for us? We need to define instructions. We need to define rules behaviors in an agent MD file. We need to provide capabilities or context and skills MD and we need to own the evils. So all of the heavy lifting, the infrastructure management, all of the same code which probably every one of us has written of us here like 20 times is no longer needed. And you can start really building your product instead of like needing to rewrite the same code over and over again.

And very important is like, hey, that's great, but what about extending? And I think looking into how extending previous agents to like those new agents work. It's very obvious that previously if we want to do like some kind of security scanning on a pull request, we would need to define or write a Python function. We would need to understand okay which CLI tools do we need to use? We need to define a new function schema and then we needed to add it to our tools need to run it and then so there's a lot of things we need to do on on agents powered by files.

We write a skills MD file maybe with some additional information on which CLI tool to use or maybe provide the CLI tool inside the environment and then we extended the capabilities. we don't need to change our code. We just provide more files to the agent and the agent decides on what we want to do. And I like to bring up some very good examples. So at a engineer in Europe, Cursor did a great talk on how they replaced uh roughly 12,000 lines of TypeScript code with a 200 lines agent files to create something similar.

So they had a very hard-coded code um orchestration for doing git work trees and they were m able to replace it with just a skill and markdown files and there are more I would say bitter lessons of ancient engineering manos has refactored their harness five times in six months last year langen has rearchitected their open deep research three times a year and then also worsel has removed 80% of their tools to achieve fewer steps faster responses and better accuracy so there's an obvious trend that with better model capabilities, we can remove orchestration code.

But if your harness is getting more complex as the model improves, you are most likely overengineering your harness. So if you struggle with model improvements and adding new capabilities which lead to more complexity and more code, you might need to rethink a little bit on how your agent harness looks. And so where does it end up? Agents are just files. We write markdown files to extend capabilities. Agents can learn from those um can create their own files.

So if you have a session and tell the agent to remember something to take notes of rules of preferences, the agent just writes it to this and then can reuse it in the later session and you can also externalize context. So if you have a very long running session and during that session you notice hey maybe I want to additionally work on another feature you can like just write that information that hand off to a file and like tell the agent to later pick it up.

Uh so what are the takeaways? We should not fight the model like we should stop micromanaging the execution paths provide general tools to the agent and let the model explore reason and discover the right solution. Own what is yours meaning focus on your domain instructions. Focus on the workflows. Focus especially on the evals, define clean tools and verify the outcomes and really build to delete. Like we have seen in the past many many times, the better the model get, the more code we can remove and the more things we need to change and obviously we all want to benefit from better models.

So what the things for you to get to do on Monday, you can scan that QR code which brings you directly to EI studio where you can immediately try out the anti-gravity harness. So you can already start prompting it. it will start your own custom sandbox. If not, um, start or create your API key. We are currently working on a free tier for the API. So hopefully you can start exploring faster soon and then definitely start building files and skills.

And that's it. Thank you for for coming. [applause] >> [music]