Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase transcript

Hello everyone. Uh is everyone excited for the the conference? Yes. Awesome. We got a full house here. Uh it's very I'm very glad to be here, very honored uh to be giving uh the opening or one of the opening workshops. Uh today if you've noticed already, uh the title is slightly different uh from what we have in in the schedule. I've basically done a rebrand, but the theme for the workshop will remain the same. Uh we went from skill issue to level up your skills.

I've decided to move the skill issue title to to with the keynote that I'm giving uh tomorrow. Uh if you you'll have time to to know more about what the keynote is going to be about. Uh but mainly uh it's um this workshop uh basically this workshop is what I've been doing in the last 2 months uh at Superbase writing our own skills, and tomorrow I'm going to present how we put this actually into production and the lessons we've learned.

Um so for everyone who's been paying um closer attention, you probably noticed that I'm I'm running this slide deck on localhost. Some of you have already noticed. Uh this is no coincidence at all. I base uh I essentially vibe coded the presentation. So if you see something off, uh it was not my fault. It was Claude. Uh but for you for you to if you don't believe me, you can see that uh you cannot do or you you'll have to be a very uh Google Slides guru to have a dark mode enabled.

So honestly, I like this layout better. So I think we're going with dark mode here. Um if there's any uh light mode fans or there are the majority of the room is light mode, I'm happy to switch back. Uh but for now, let's go with the with this one. So, to to do a a little presentation of myself before starting the workshop, my name is Pedro. I'm I'm from Portugal, Lisbon, and I work at Superbase as an AI tooling engineer.

Essentially, my day-to-day is to think of how we can make the Superbase the most agentic friendly as possible and improve the agent experience. So, we you've probably heard about develop development experience, the DX. We're more focused on the AX, which is the same thing but for agents. In this workshop, we're going to talk a bit about skills cuz essentially that's how we've been improving the the performance of agents around a product like Superbase or company like Superbase has multiple products.

The secret sauce has been basically skills. So, we're going to dive into how to write one, how to test it man first manually, and then how to automate the testing with evaluations. So, to start with, how many of you have heard about skills? All right. So, almost anyone everyone. So, what I'm going to say it's probably no news to you. Skills are basically folders with instructions and files for you to to run workflows, repeated workflows, or give custom um essentially custom information to your agents or provides a new set of of tools, let's say, in form of scripts.

So, there's a a bit of a misconception about skills. Usually, the skill.md, the main file, takes the the the spotlight, but skills can actually be more than and the main file, right? So, the the main file is basically a markdown file named skill.md where the essentially the the main information about the the skill lives. It is composed by this front matter at the top which essentially has can have multiple fields but the two required ones are the name which basically identifies the skill and then the description which tells the agent what the skill does.

The main What exactly the the the the skills basically bring that tools like MCP didn't was this concept of progressive disclosure. Progressive disclosure is basically when the agent or the all the information about a subject is not loaded straight to context. Instead, you just load the exact amounts of information that allows the agent to to choose to load the rest of the information once it actually needs it. So, in this case um the the skill.md file is designed like this.

So, the front matter will be loaded at first to the to the context of the agent, not the content of the the file. This works as an envelope so the agent has knows from the description what the skill does and when it should load the the rest of So, when should it look for the information inside of the file. Inside this file, you can also reference another files. Usually these other files are either markdown files or script bash, python, whatever you you would like to to to reference.

Starting on the reference files, you usually put them inside a reference folder and they provide more information. You can think about a skill in this format as a book. The skill.md you can think of it as the index on steroids because besides of having these links to the other files, you can think of them as the pages of the book or the other chapters, uh, you you'll have um, custom information you can have custom informations and then also reference it to to the other files.

The reference files they have nothing special about them. They're basically like the a normal regular markdown file. You can think of a similar to skill that I'm the file but instead of being the main one it's the one that uh, that reference. Uh, you can also funny enough you can also reference a files inside of reference files. So you can make basically a graph out of out of a skill. And for for scripts, I've actually talked about how the our MCP and skills differ from each other and we're basically comparing apples to uh, apples to oranges when it comes to MCP and skills.

One of the misconceptions currently probably was already the bunked the debate now it's more about MCP versus CLI. But when the the skills were released back in I think it was November or October last year, they basically started this debate about well it's should we use them instead of MCP because if if I can run if I can provide more information more context to the to the agent without actually loading every tool to to the context like the like the MCP and I can also have screen so I can have actions just like I have on MCP tools should we use them.

And the answer is you should use both to be honest. If you're building anything that it's an integration you should use MCP, right? Anything that if your agent doesn't have access to bash you should you should use MCP to integrate to your service. Skills actually just provide more context to to agent, right? And you can define workflows everything that you would not that that you don't have space to define on on the MCP tools.

Uh descriptions, you can define them on skills. Um also regarding the comparison the the debate between skills, scripts, and the skill and the MCP tools, the main difference is that tools don't need um an environment to to run. Uh the agent can just call a tool, knows how to call a tool, especially if the uh the MCP server is remote, and the tool will run on on server side. Well, the the scripts uh well, they basically are loaded into your machine.

They run on your local environment. Uh and they're tied to the whatever environment that you have. So, if you're running on Linux, they have to be Linux compatible. You're running on Mac OS, the same. Windows, I'm not going I'm going to even start it about it. Uh but essentially, that those are the main differences between the MCP tools and the the scripts. Uh oh, is everything clear? If you have any doubts, uh feel free to I'm going to have a a little demonstration.

This workshop is going to be more more of a walk-through than actually code along, but feel free to to tag in. Um I have a a GitHub repo uh prepared, so you'll be able to visit it and to explore it. Um but if you have any doubts in any moment of the of the workshop, uh feel free to to interrupt me or to raise your your hands. So, moving to See, this Exactly. This is So, I tested this on a smaller screen, was working.

You can see it was live coded. So, uh how do you test your skills, right? So, if this is just a markdown, how do you test your markdown files, basically? Um so, to test a an applica- a a piece of code, it's uh it's already it's straightforward, right? You we already know you have all sort of um tests types. You have unit tests, integration tests, so you can test the whole flow, what do we call end-to-end testing. Um well, it essentially, when you're testing a Markdown file, you can basically do exactly the same.

You can be as granular if you uh as you you want. Um but usually, since we have an LLM in the loop, you'll have something called evaluations. So, uh for those of you haven't heard about evaluations, or evals for short, um they essentially um are a more um a non-deterministic way of testing the output or the behavior of an agent or a model. Uh you can test both an LLM or an agent uh with with evals. Uh essentially, you'll be the most common structure I'm going to be to to to present to you at the end um a framework for you to test your your evals, like a very simple one where you can start, and I'm going to uh dive uh deeper on evaluations there, but essentially, um they usually are uh um made of an input, an expected output, just like a regular test, and in between, you can um evaluate the steps that the agent took, the reasoning, uh the tools that it that it called, uh which is uh normally more interesting and easy to to evaluate the than just like uh a rejects on the exact output, since this is non-deterministic.

So, there's essentially uh a framework uh to that you can follow to test your skills. Uh this one was proposed by OpenAI on their system on their blog post called Systematically Evaluate uh Agent Skills. I think they released this back in January or February, so not that long ago, but all this is fairly new, so this is basically prehistory. Um so, you start uh by defining your metrics, so what you want to evaluate on your skills.

If you're building a skill for your product, for example, what exactly do you want the skill to to highlight to your to your agent? It's going to be to forward it to the documentation. Are you putting some specific instructions, specific workflow? So, depending on what you want to to evaluate, you start this eval driven development. So, this test driven development, you start by defining the metrics. What you what exactly good means when it comes to the skill.

Then you create the skill itself, right? So, you write the skill.md file. So, you write the the skill.md file, any scripts alongside it, the reference files if you want to. They're all optional. The one you the only only required is the skill.md file. And then you you went the you move to the testing part. So, you run the evaluations or or you run it manually. Um, I've recently heard the the the CEO of Braintrust during the podcast podcast.

I don't know how many of you know Braintrust? Okay. Not not as much the not as many as the not as popular as skills, but so Braintrust for the those of you who don't know Braintrust, it's a platform that allows you to systematically run evals and provide you like the full picture of um of the of the the agent behavior during the the evaluation scenario, right? Um, I'm trying to think about another platform to compare with, but this is fairly new to be honest.

Um, so you can think of it as like an observability tool to to check your the behavior of your of your agent during a specific control scenario which are the evaluations. Um, so you move to to the testing part. Basically, you run a set of evaluations scenarios. How you these are defined by the input and expected output tools that should be called. So basically, how do you expect your agent to behave? Um, and then you move to the grading part.

So how did the the agent do? Well, essentially it's this is very similar to a testing cycle, right? But now we instead of having a deterministic output, you can have it's non-deterministic. It's an LLM in between. But you can still have deterministic parts to evaluate on. And then you iterate basically and repeat. This is the that's why this it's a it's a cycle. Pretty similar to any of the the the test development cycles that we that we have at the moment.

All right, so jumping straight to what we're going to do during this workshop. So we're going to write a a skill. I've prepared a little demonstration app, a demo app. It's going to be a performance review application with four, I believe, four employees. One employee, two managers, and one HR um representative. And essentially, we're going to there's some errors on the database side that we're going to find and fix.

We're going to build a skill to help the to guide the agents to fix them, all right? And then at the end, I have as I said a framework to test automatically the the same scenario that we're going to test manually using using a vault. Before moving to the demonstration, how many of you have have heard about or used Superbase? All right, so almost almost anyone knows or used Superbase. I've I've seen some some hands down, so still I'm going to give a little brief.

So Superbase is essentially a back end as a service. You you can think of it as the open source version of of um I got I got I just can't Thank you, Firebase. I only fire ship was coming to to my mind, sorry. To Firebase. And if you don't know Firebase, you're probably living under a rock. No, but it's actually it's the back end as a service. You can use it to build any back end as you as you would like coming straight out of the of the box.

We provide a database for for you to just plug into your application and run on Postgres, one of the most if not the the most popular open source solution out there for databases. You can easily integrate including authentication on your application running a storage to save files um and many other things edge functions we which are a lambda functions for those of you come from the AWS environment and so forth. So the demo application that I've built was built on top of Superbase, of course.

And so you can follow along. Here is the QR code. At the back anyone everyone scan scan the the QR code. So I should I make this bigger? Bigger, right? This. Okay, just so everyone can see. I'm basically editing the the presentation at the moment as we speak. Let's see what clothes has to to offer us. Bigger? This is the cool thing of of I putting our presentations. I really recommend. Uh I probably spent the the same or or more time than than if I just use something like Google Slides, but at least it's more fun.

Uh and Entropic should be thrilled about it, for sure. All right. Let me know once everyone is in the Is that the can see the repo? If you cannot see or scan the QR code, uh I should probably make the the link bigger as well. So, you asked for a demo. Here's the demo on my slides. Uh So, mainly everyone you here and this room used uh Skills, so I probably won't have to to sell you uh the the the power of Skills. Uh but if you if you're still a bit skeptical about Skills, this whole presentation with without Skills, this whole presentation will be a lot um not pleasant, let's say.

Be much uglier in a sense. Okay, you should probably see it now. So, basically navigates to uh to GitHub Rodrigues PN, which is my um nickname, and improve skills workshop AIEE Europe. It's a very long name. All right. So, is everyone at the at the at the GitHub repo at the moment? Okay. Everyone was had no trouble. All right. So, uh not this one. Right. So, this is the repo that you should be looking at. Uh essentially, it's I know that it's a big repo, but we'll go we're going to break it down.

Um I'm actually going to move. Okay. I'm going to I'm going to move to VS Code. All right. So, here we have uh we have an two Next.js apps. Actually, the the slides are also embedded here. Uh the the Next.js app that um that matters, it's inside demo, right? And to give you an insight of what that looks like, it's basically this. So, it's a very simple application. It's You can see that it's a vibe-coded application, to be honest.

The the layout it has nothing special on it. Uh you have, as I've described earlier, several um employees of this fictional company. Um and you can think of it like a an internet or a performance review application where uh you have all the information as an HR um employee, you have all the information all the information about the um about you the the other uh employees of the the company. Uh and you can change, for the sake of the presentation, you can change between uh users, right?

So, what we're going to do uh is first without a skill, we're going to try to implement a new um a new new view uh here. We're going to implement a reports view. Essentially, this uh reports uh part of the um of the application is going to be uh is going to show uh both the salary uh and the the average rating for the performance review of each department. So, uh so, HR can be um can know what's uh uh can have like an overview of the of the whole company.

Um So, before we start to vibe codes, cuz um during this present the this workshops, no one actually writes code anymore. Uh so, of course, I'm going to vibe code it. We're going to just break this um a um this application down. So, if we navigate to the dashboard, it's uh nothing special to to see. You have It's basically the the page the first page the main page that you which you've seen. And then you have here the the reports where we should have uh Yeah.

Where you should have this set view exists. That is going to um So, I've I prepared the back end. We're just going to to to create on the back end the view as a SQL view on the the back end on the on the database and then we should be able to to see to see it on on the application. So, I've prepared Where is it? Yeah. So, I've prepared the prompt. We're going to live test it. Fingers crossed that this work. Wait. Uh first, let me navigate to It's app.

Okay. Let's see where we have more control. All right. So, essentially we're the ones in the back. Uh I'm just I'm going to ask Claude to create a department stats view that shows the head count and the average salary broken down by department, all right? So, for HR to have a full overview of what's going on in the the company. So, we're going to eat the prompt and wait it wait to see what it it should come up with. Uh Right.

Um forgot about this part. I have um this MCP server configured. If you have uh you actually have I totally jumped uh the read me. Uh if you're follow along, sorry about this. If you follow along if you're following along, you can follow the setup guide to to get a to get your application started locally. Uh it's essentially is going to install the the dependency clone the repo, install the dependencies, start locally your your Superbase project.

You don't have to have the CLI installed and we're using NPX to start to run it as a um as a binary Um just resets the the database state so you start from from scratch with the seeded data and then just run the the app as as the running NPM run dev should be available on localhost 3000/dashboard. You also have you'll have this MCP .json file prepared. This essentially is pointing to the MCP server that we we Superbase enable for local projects.

No authentication required so your agent should be able to just load it on demand. This this MCP server exposed a set of tools is a I don't know many of you have used the the Superbase MCP server but we I think we are currently have something along 20 20 something 29 tools I believe for the production one. This one is a smaller version has 20 tools but you can basically perform essentially almost any anything that the the the one to connect to your remote project does basically list the the tables that you have execute SQL the straight on your database apply migration and run the database advisors and so forth.

So essentially what I what if I started to do was to list my tables. So I've asked for a view is going to review the the schema that I already have implemented. So I'll let you And now it's going to run the apply migration tool to create the view. So it's basically doing a schema change on my database and it's going to create the the view. If we inspect the view we're basically creating Um, or replace a view of departments stats, the name that we gave.

Uh, and we're by um, fetching all the information uh, from I think department exactly uh, from No, from profiles exactly and group by department. Okay? Whoop. Made a mistake, going to try again. Okay, let's going to test it. Actually, something that I really like about All right, then here's the here's our uh, view on the database. So, uh, we currently have it on the database. Let's see if that's also enables on the app.

Mhm. Okay, it's not. Then, let's quickly uh, I edited SQL you the name I gave. This is the This is essentially the the problem with live views. As you usually doesn't go well at first try. Uh, with I'm going to repeat uh, And let's see if it if it implements. If not, we can just run the the SQL query for you to to see as different users uh, to for you to see if everything is working um, accordingly. So, for now is going to to implement on the next JS application so we can have a nice interface to to check the results.

Um I need to enable everything. I'm just going to put on auto mode so I can continue to talk. So, essentially the the agent created the the view. Tested. Said everything is working accordingly. We should not the the app was the the feature was implemented. All goods. We're actually going to see if he's actually if everything is it's good or not. Um so, let's give it a Let's give him some space not to pressure not to pressure it to create the the the feature.

Let's just wait a bit more time. In the meantime, if you're following along, you can also play with it. Uh change the the layout. I'm actually using the using cloud code. I don't know just doing the a brief survey here in the during the workshop. How many of you are using cloud code as well? Oh, fairly almost everyone. Okay. Um How many of you are using cursor? With cloud code or with a plugin or Okay, yeah. At least one person.

Uh we're going to have uh some um cursor folks here. I think from Anthropic as well. Uh OpenAI is going to be here as Gemini. Of course, Google DeepMind is sponsoring the event. So, um we're basically going to have the whole gang here. Uh okay. So, we should be I'm trusting his word. All right. So, it says that we should now be have correctly displayed the the department stats are so let's see if that's actually true.

It looks like it. Yeah, so we now have this uh cards with the whole view of the company. So I'm logging in as Julia from HR. We can see that we have five people on the engineering team with an average salary of 107k. HR as well will only one person will would be Julia and product has four people and that average salary. So so far so good. Looks looks okay. Let's see. Uh so this is like this is sensible information the the the reports right so uh we're expecting that the other um the other employees will not have access to it and even the managers only have for their departments.

Let's see if that's the case. So let's navigate to Bob. Bob is the head of engineering. Oh. Okay, so so Bob also can see the the performance reviews of both the information of both HR and product. Well, it's not that it's not that bad, right? It's it's not ideal, but at least he's a manager, right? So he should be access to privileged information anyway and who doesn't like a transparent company? Let's see if our hell this is okay.

Okay, this is this is problematic. So we basically created a view. Claude said everything is working because as you can see the information is here. It was created. But it missed something that is training data basically missed something which was for Postgres specifically when you create a a new view the and your table has row level security enabled. So for those of you that know, a row level security allows to for you to define who can see the information on a on a specific row on a database level.

So, without trusting the the application, you can filter it directly on the on the database. So, in this case, we should be limiting the the the view of of the rows uh by user ID, right? And the user role. So, if a user has an employee um and as an employee role, it should not be should not have access to the rows of the that don't belong to the to them, right? Uh we have row level security enabled. If you navigate to our Superbase um migrations, you can see uh row level that we have row level security enabled both uh on profiles and on performance reviews, right?

And on a performance review, should be should be about right uh right? So, we have reviewer ID equal current setting. So, it should work. Why is it not working? Well, when you create a view on Postgres, um by default, the the the permission um it creates with the permissions of or or the the credentials of the the user that created the view and not uh with the the the credentials of uh of the of the table, let's say, with the the row level security.

So, basically, by default, it bypasses the row level security uh that you have in place on your that you might have in place already on your um uh on your uh on your table. So, for for this to not to happen, we have to add a security invoker. Uh, we have to use a security invoker flag to transfer the row-level security policies or to enable the the the RLS policies on the view itself. So, this is the why currently everyone can see everyone's because the the row-level security policies were basically bypassed on the on the view.

So, uh, for the sake of the of the demonstration of this workshop, I've already created um, a skill. Uh, uh, I prepared the skill uh, for for the presentation. Uh, and essentially the skill is three main main security points about Postgres uh, for that the engine should be aware of during the the presentation. For this one specifically, I actually over fitted to the exact view that we're creating, but models right now are smart enough to uh, generalize this and if I wanted to to create a new a view, you will be able to.

Uh, essentially it has to create with this flag. Uh, since Postgres uh, version 15, this flag was enabled and every time it's enabled the the row-level the RLS policies uh, uh, are also enabled on the on the view. Um, as you can see, it's um, it's actually quite human readable documents. Um, mo- most of you have already written skills, so I'm not going to uh, dive uh, deep into into this. Um, but as you can see, we have both the title.

Uh, let me just move this. We have the title. I called it Superbase Security uh, and the description uh, uses the the verb use. This is an insight that I that I got from some experiments that I did using verbs, mainly the the verb use, uh increases the chances of the skill being loaded, um at least on Claude. I don't know if this is default behavior for for Claude, if he was trained to recognize to more easily recognize verbs, it's actually use, but I found it more efficient to if you write use and then the whole purpose of the the skill in front of it.

And then a regular markdown list we have the the view case there, but also another checklist points for security um on RLS. So, public schemas should should have RLS enabled by default. Public schemas or exposed schemas are the the database schemas that are going to provide information for the the application that the user can see. So, for example, the the users table, the profiles, the performance reviews, all this information is going to be fetched by the front end.

It's completely secured cuz Supabase makes it secure by by allowing you to to fetch information on from the front end. Uh but the the key part here is that if you don't enable row level security, you will not have this filter on the on the table and you will have to rely on the application logic to to make the the filter. So, enabling row level security at least makes it safer for you as the back end engineer that the you only expose the information that you actually want from the start.

And then a couple more things that I'm not I'm into. So, if we we can install this skill on this project by running Where where do I have the command? NPX, yeah. Whoop. So, I'll be Oh. I'll be using Where's the skill? Okay. I'll be using Vercel's NPM package called skills. Curious to to know how you guys have been packaging your skills. Have been Have you ever used this this package? Are you using plugins? Just This one.

Yeah, this one mainly. Yeah, it became very popular a few months ago. Yeah, you can install it both globally and and on your project. Yeah, and also for multiple as support for multiple agents. While plugins for now are still tied to the agent that is going to load them. So, Cursor has plugins, cloud code has plugins. I think other vendors have as well, but they're specifically distributed and made for those specific models.

So, we're using this one to install. You can install any skill from a repo online that has a skill.md file or or you can use it to install the one locally. It will auto detect the the location that you're trying to fetch from based on the format. So, in this case, we don't have any GitHub, we don't have any HTTP protocol there. So, we have a dot slash. So, it'll recognize that it's a uh, it's a local one. and for this, I'm going to uh, uh, move to the main yeah, guy.

And on a good, old-fashioned way, going to run on the bash. So, it's going to pop pop this up, um, ask me which agent do I want to install on. I'm using Cloud Code, so I'm going to install it on Cloud Code. If you're using any other, uh, any other, um, agent Arnes, you can also install it as long as it's, um, uh, it supports it. Uh, I'm going to install it on the on the project level, so it's going to, uh, in this case, going to create a dot agent folder with the skill and, uh, link it to my dot cloud uh, slash, uh, skills uh, folder as well, so this so Cloud knows how to where to find them.

I'm going to simlink, and we're ready to install. So, we if we Let's not expose my key. Uh, I'll delete this just for the workshop, so I'll delete it afterwards. Feel free to use my my free credit, uh, for the time being. But, essentially, created the, the, uh, uh, dot agents uh, Where is it? Yeah. So, I also have some more things that we're going to see afterwards, uh, but the essential part it has the skill that I've showed previously.

Yeah, there it is. It's the the skill, and it also created a sim uh, a simlink, a symbolic link to the dot cloud. Uh, this is how the package works, uh, and this this way allows to to Cloud to either search on dot agents, which is becoming the standard, or on the dot cloud, uh, folder that it has. So, let's see, let's run the same prompt again on a new session. Um, let me go back to the apps demo. Yeah. And start a new session.

We should have this one enabled. Yeah, there it is. So, Claude's is aware of uh the Superbase security skill now. For you to run skills, uh you can either um just run your prompt and uh pray that uh Claude's imports your skill uh based on the description that you gave. Uh you can uh includes the keywords use and then the name of the the skill that you have on the prompt, and this will uh almost 100% of the times load your skill.

Or, if you're using Claude's code, you can just slash and write the name of of your skill. And this 100% guarantees that Claude will going to import the skill. So, for for our use case, uh for the presentation, I'm going to cuz I cannot afford that it doesn't load the the the skill. Let's Wait, I need to uh I need to reset the the database to create the the view again. Um to be in the workshop. And it's NPX Superbase DB reset.

Yes. I'm just resetting the the database, applying the migrations from the from the start. Uh I didn't It didn't create any um any a migration file. I He applied direct the the migration directly to to the database. So, we now should should be good to to go. It's going to uh bring down the the database and create a new one based on the schema that we defined on the migration files and the seeded data. Yes. Yeah, that's a that's a fair point.

So, that the your whole question or observation is that the initial promise of skills they were presented by Anthropic were Yeah, uh so since this is this is like on the the agent side, right? The agent decides when to load this. The best thing that you can do without explicitly either with the slash command or the use and then the name of the skill on your prompt is for you to play and play around with the description uh and run a bunch of tests either manually or or automatically to to check what actually works and not for the the ones that are that you're expecting the agent to behave, right?

So, you define a bunch of scenarios where you think that the skill should be loaded and when the skills shouldn't be loaded. Um you test it out. You can test it out on your machine like on this scenario, I don't want the the skill to to be loaded prompt the the the prompt on on cloud code, let's say, and check if the skill was loaded or not through the CLI. Um and then play around with the description to see what actually works or not.

Like this without actually explicitly call the skill. This is the best thing that you can do to to to to test uh if the the skill is being loaded correctly or not. Yeah, it's we're still at the at the very beginning of uh like a a very early stage uh of of skills. Even for MCP like all all these um agent stuff, it's fairly new. So, we're still we're still standardizing things. Uh we're still figuring out what works and what doesn't.

Progressive disclosure was something that no one was talking about uh 6 months ago and now the it's fairly it's fair to say that it's one of the north stars of uh agents uh development. Uh so, in 6 months from now probably could be another thing. So, or skills could be the standard or maybe Anthropic or OpenAI or someone else found a more efficient way to manage the context or provide more context to the to the agent.

Uh so, we'll see basically. All right, so the database was was reset. Okay. So, at least now we have the view, but we don't have the information on your database. So, now we should be able to run the same prompt again, but we but with the with the skill. So, if if we edit the prompt, you see it was quite fast, I don't think. Yeah, but it didn't create one. Okay, let me try another thing instead of Instead of this, let's use to create.

Let's see if it works now. Yeah, okay. So, it loaded the skill. So, now at least should have the context to create that the the RLS or the security invoker flag should be included when creating the view. And the steps should the the the rest of the workflow should remain the same. So, it will list my tables, right? Exactly identified the tables. And now if we look if we look closely, we can see that we are we now have this the the flag here is going to be on the on the migration.

So, let's see if with the flag this is the expected result. This is what what happens when you wipe code a CLI, you know, I have the the UI duplicated. Right, so it created the view. We should be able to see it. But Alice shouldn't. So, what's happening? Wait, okay. So, do I have to reset? No. Hm. Interesting. Should have another. Uh probably Let me just see if I have it uh here. Uh where did I put it? Let me search on the account.

I'm going to cheat here. Let's say the both and the employee be able to see information to see the at and the bonus Let's take control of the Okay. Basically life troubleshooting. Why is not happening? Probably from a different a different policy that I've defined here. Uh But now it's going to troubleshoot. Let's see if the the skill actually improves the the efforts here. If not, I have something on my sleeve cuz if you're not aware of a Superbase basically has um database advisers that you can use to try to identify early early on identify some potential vulnerabilities or schemas that are exposed information that might be exposed before you're running into production.

Um So, if if you can't figure out by itself I'm going to include on the skill to also run the advisers to to check. So, this is the the main part of skills is that you can Oh, that's You can see well, it's a It's a very poorly written application. Let me say. Uh It's essentially the the main part of skills. It's not if if this specific demo works or not is that you the behavior changed once the it loaded the skill, right?

It created with the security in vocal parts. Uh and with with that that they just shows how powerful it is that that you can create you can change the the behavior or or guide the the agent on demand basic based on the on information that you that you put. You can think of the skill that MDS is a prompt template that you can give to to your agent. So, let's just quickly troubleshoot. Oh, it's even offering to apply a migration.

Let's see if it doesn't break my my app. All right. So, it seems too complicate too complex. Anyway, going to going to move. What's that? The separate table. That's it. HR Anyway. Could could you look at the context like slash context in your just to see like how your context looks? Good point. I have a fairly amount of skills as you as you can see I've been playing around with them. I also have the some of the pre-install MCP servers for that um that Superbase enables.

Uh But essentially skill it would be more interesting if you if I've just um if I've compared the the context from before and after loading the skill. So, right now skills take 1.3 thousand tokens on my context, right? As As you saw, I have more than than just this one skill. But, the skill was loaded, so the whole information inside skills.md was loaded to to context. If we clear and run the context again, the the skill amounts.

So, this skill is not it's not enough to for you to see. But, as you can see, the the skills take quite um less space uh that that the MCP would uh from it's All right. Oh, okay. I have a newer version of the cloud code. So, for those of you who are not aware of this, uh Anthropic recently released the tool the the tool search tool, uh which is a mechanism for for cloud code to load tools on demand. So, it doesn't load basically progressive disclosure but for MCP tools, right?

Um the the main difference between this progressive disclosure or the the tool search tool um on on cloud code and skills is that the progressive disclosure is uh built by design uh for skills. So, it's like already baked into the structure of the instance of the skill, while on MCP it's still not a standard for all tools. So, it works for cloud code, but for many other clients, it won't. It will just load all tools straight to your context.

So, um this is a for now thing for just um for for just cloud code. If you're interested about it, uh we're going to have the the founder or one of the co-founders of the MC of MCP speaking on the 10th, so on Friday, is going to give a brief overview of the the MCP roadmap, um, which if it's something if anything if nothing changed since last week when he presented this in New York on the MCP Dev Summit, he should bring this this progressive disclosure part to the tools to bring it to the protocol itself.

So, yes. Um, let's say that we have a very large database and uh we have to load in the context the schema of this database because we we have to query the database using agents. Mhm. Okay. In your opinion, is it better to use a skill or um an MCP or something for a to to load this schema but progressively? Okay. Uh Is it possible to use this schema to progressively disclo um load the schema of this big database? In your experience.

Yeah. Yes. Okay. So, is your question more about how should we access it or the whole architecture of this pipeline to import the the data progressively? >> to to ask to a an agent to to query the database and obviously the agent must know the the schema of the database before or not. I would How can you teach the the agent to to query the database? Mhm. Using the skills, using the an MCP server or something like that.

Okay. And if you use the skills if you decide to use the skills to to load the contacts of the agents with the schema of the database, is it possible to progressively load the schema within the context? Mhm. Okay. Got you. Um so, let me break let me break the the situation let me break down the situation for you. Here you you'll have um essentially two parts. One is what's going to be on the the context or what's going to be loaded and the the specific information that you want to to have on your on your scenario.

And the second part is the actual mechanism the the extraction mechanism that you're going to use to load the information from the database. So, for the second part, to to load the the information from the the database, you can either use a script so a skill that invokes a script or an MCP tool. Um I would advise to use an MCP tool because you can use it if if you're using on production or on remote project, you don't rely on your local environment, you don't have to manage the keys, and the tool it's already standardized and you already have the the authentication baked into the protocol.

So, the agent never manage the the application token. It's on the it just runs the tool and and it works for the for it. To progressive disclosure, the the information on the database, it will you'll have to um you can include it on on a skill. Yeah, you'll be using the MCP tool. So, on the skill you'll probably state that use this tool to load and in the tool implementation you have to enable it to not load to progressively load it, right?

So, to load into chunks. Um it might be just enough from the the the tool parameters. The agent should figure it out by itself that if you put a parameter called buffer, for example, should be able to load it in chunks, right? Instead of the whole table, but if you want to have 100% sure that it's going to load into chunks and use it properly, I would also package we with a skill and describe it how our intent to to use this uh uh this tool.

So, this is actually how both skills and MCP play along together. It's the the tool to enable this connection, this integration, and the skill to describe how to use it. Yeah, this is how I I would implement this this type of system. Uh thank you for for the for the question uh and it got me the opportunity to to basically talk about the how how to use both skills and MCP and not put it uh uh one against each other. Um so, now as I promised, we should be moving on.

Um I'll have to give it more time uh to to figure out because I've I've basically during the the workshop when I was preparing the workshop, I've I've gave it a bunch of vulnerabilities. So, if I just kept it simple and that one, the demo should would probably work. Um since I have more vulnerabilities exposed that uh if I had time, I would um try to solve it. Uh It didn't for for the moment, but uh uh but you you saw on both uh scenarios that the first one didn't have the security uh flag, security invoker flag, and the second one had.

So, at least we can um in we we can imply that the the the skill was doing something. It did the agent saw the information on the skill. It It merged with the system prompt or stored it near near the system prompt and change the behavior accordingly. To test this, so if you want to move this this part the the skill into production, right? So it works on your machine. It's a it's a tale older than time that it's working on my machine, but I don't know if it's going to work on your agents on your machine on your environment.

So to have this to test this or to automate this testing and with this we can unlock having a pipeline, for example, if you change one thing on your skill, how can you reliably tell that it's it keeps doing what you're expecting didn't break the previous flow. So if I change one of the checklists, how can I ensure that the the other ones were still working, right? So for the this is where evals could step in. So uh evaluations it's a very broad term.

You can basically evaluate anything. Since this is a markdown file, it's a free text file, you can evaluate basically anything. So it's fairly difficult for you to the most difficult part to create evals, I would say is actually coming up with the scenarios cuz you would first have to to know what's the expected behavior of your of your agent. Um so coming up with representative actually goods scenarios that represent a fairly amount that cover fairly amount of use cases that you want to to build are the most difficult.

Um and there's still not a standardized structure to create evaluations. You can use you can test it by importing a bunch of prompts and expected output from a CSV file, from a JSON file. You can use tools like Brain Trust or LangFuse to test it and to to have a an analytics and an observability layer on top of it. For this presentation, I followed the I followed the what Agent Skills Open Standard defines as to to design the test cases.

So, if you're not aware of this website, this is the landing page of the Agent Skills Open Standard to try to standardize what a skill is and how should behave. And they basically propose a very simple structure local way to test the the skills organized by you'll have an eval.json that essentially has a set of evals, so an array of of eval scenarios. You'll put the prompt that you're going to give the agent, the expected output from the agent.

This is only only if you have an LLM as a judge. This is a technique used for non-deterministic evaluation. You you would have Instead of a human, you can give the outputs of a of a an evaluation run to another LLM. Say it defines a success criteria and let the the LLM who's who's doing the whose role is is to judge in this case. That's why it's called LLM as a judge. To give it a grade, basically. So, this is one part that you can automate on your evaluations for non-deterministic workflows.

You can either assert if a tool was called or you can give the the results to an LLM and non-deterministically try to get the the agents to to grade the the performance of the other agents. So, basically have agents evaluating agents. Um So, I followed this this structure. I gave the same um the same input here, right? Uh so, the the agent that is going to run this evaluation is going to get the same input that we that we had.

The expected output is that the security invoker uh it's true, so it's it's present on the um on the app. Uh sorry, on the view. And I have and then I have a bunch of uh assertions that in this case um uh I'm going to check it's uh um deterministically, right? I prepared a Python script that it essentially just resets the the state of the database, so we we ensure that uh since we're running this locally and not on isolated container, like a Docker container for example, uh we have to make sure that the system is always starts uh from the same ground, so I'm going to reset the the app.

Uh if you want to to run the the evaluations as well, you have to pick your own Anthropic key, uh create copy this uh you can follow the the readme inside the the Superbase security. Uh here you'll have how to set this up. Um but then I I will run the uh the Claude code CLI on into I think it's on print mode or I don't remember what it what they called, but essentially like we I will run it as a binary headless. Um so, the agent will receive the um the the prompts that are that's on the evaluation uh as the task to perform.

And I'm also going to to give the condition. Uh we're going to test two conditions, one with the skill and another without it. And essentially so, for you to to see what are the conditions Claude's, this is where the Claude code will run. And if the condition is with skill, we're going to load the skill.md into the the system prompt, right? Um if you if you would actually like would like to mimic the behavior, you would run this on the Docker container.

You will put the agent skills uh on the dot cloud {slash} skills um directory inside of the Docker container and let organically let the cloud code uh find them and use them. Uh for for this presentation, this is a very simple setup. I've just basically appended to to the system prompt. So, So, we're going to run the evaluations. Do I have the other Yes, I do. Okay, I think we run it on the the Firebase. Oh. It's not finding the Why is it not finding the the skill Firebase?

No. Okay. Oh, yeah, I have Wait. I know what's going on. I have the the wrong name. I'll change it. All right, so we started by running with the skill. So, the first result that we should get uh is the the with the skill. It stopped. Now, it's running without it. And then, we're going to compare it. This will output a workspace iteration one um folder, and we can compare it both the outputs of with the skill and without it.

Uh while the without skill is loading, let's just quickly inspect what what the uh with skill output gave. Um and essentially, you can see that it created this uh the view with the security invoker. And then, we have this grading.json file with a bunch of information like the the assertions that we we've put on the eval we've set on the eval.json. Uh we have them here. And we can see that for this one, graded as the as failing even though that graded Where is it?

Not found. Uh the view as security setting. Oh. Okay. I'm actually evaluating something wrong. So, the problem here now it's uh with the Yeah, is it the skill? Uh view. So, since I I was expecting this to to create an PG class uh RL options instead of just inspecting the the view, it's giving me that uh it failed. But, the key part Has it finished? It has not finished. Still running. Taking a long time. Could be. Okay.

And now, we can inspect. Okay. So, uh this is actually good good insight. So, um with these results, this is the the tricky part of of writing evolves. Um So, as the as like normal tests, uh the results will depend on how you implement them, right? It's just code. Uh so, if you're evaluating something wrong or some or not the the expected behavior, you're going to have wrong results. It might not be because the the system is is not working.

So, uh we've tested manually and see that with the skill it created with the security um the the security flag, we can actually just inspect it here with the skill created. Let's see if if on this one surprisingly this time it did. It's a non-deterministic um the non-deterministic behavior of of Claude. Um But, since I was evaluating something wrong, right? Uh I was expecting it to create the or or inspecting uh the um a meta schema to check if the the the view the um the security invoker was there or not instead of just inspecting the view directly.

Um the results came a bit off. So, it said that with the skill it failed. And with the um without the skill it passed. So, and if if we inspect the both outputs, they're basically the same. So, we did this just to show you how tricky it is to to write evolves because it's although this can happen on with the with the regular tests, um it's easier to catch because the the output is deterministic, right? It's just code.

Um here, if you're handing to to an LLM to evaluate, it can all uh sometimes hallucinate. So, to finish, uh cuz we're also almost running out of time. To sum up the the structure, this is the one that they recommend. Uh I find it very easy to implement to do it to getting start with. Uh later on you can move on to more um complex uh evaluation scenarios like running on a Docker or in a sandbox uh to guarantee that uh you get the fresh environment uh with just one skill that you're testing on your set.

Um But essentially you would just put two conditions with and without the skill, compare the results, and see uh run run them on the harness the agent harness that you would like, and you compare the results uh out there. This is basically your very first uh evaluation pipeline to to test a skill automatically. From my end, that's all. I hope you find you found this uh workshop useful to to uh get your skills leveled up and ready to productions.

Um going As I said in the beginning, I'm going to give uh a keynote tomorrow a keynote no a talk tomorrow about how we've implemented uh uh and created the the Superbase skill for the product itself, how we're keeping it main maintainable while ensuring that provides value, and how we're uh testing it into production. THANK YOU. ANYONE HAS UH ANY DOUBTS, QUESTIONS? I'LL I'll also be yeah. Uh so, I have a question about uh like the number of skills that you typically install on your environment because with this progressive disclosure, it seems like we can basically keep adding different skills, and the agent will automatically um basically the agents will automatically find them.

Uh do you have any recommendation on how many skills to have or is there any limit or we should just basically keep adding? And it will magically work. I'm probably not the best person to talk about this cuz it's easy for you to get into this rabbit hole of just like especially when you're experimenting getting a bunch of skills as you saw I had a plenty of them installed globally and I think it's fair to say that I don't use them all on a daily basis.

But it depends if if you're using them on your local machine, I think it's pretty be pretty easy for you to get this messy environment where all of them installed or most of them installed. For look for your local environment, I wouldn't for now since it's very experiment on my personal opinion, I would not constrain myself on like space management or context management about this. The progressive disclosure it's a very powerful thing that you can explore in this case.

You sure if you have skills that you don't use you're going to have them fill your context window but the descriptions are so small that you can afford to not delete them if you don't want to. Into production, treat them as any artifact that you would have on your CI. So keep it clean into production into your CI. I would keep them only the the exact skills that you're that you're using in that specific case. Another piece of information that I could give you on the production part is that it's now more and more common for you to also export skills or make skills available on your repos as like a piece of documentation.

So, treat it Treat skills that you put into production as actual document as you would read documentation. So, it's important for you to keep them updated, included on your include the the updates workflow on your cloud R&D or on your agent's R&D. So, you make sure that if anything changes, you will change this the skill as well, like you would do on on the documentation. If a feature or workflow changes, E, from time to time you can also create a a job to to check if the skill is still running a fair workflow.

If somehow you could check if the the skill have been loaded by your users. E, in a if if it haven't been loaded by a by your users for a long time, does it still make sense to have it there? So, yeah, this is basically the the piece of advice that I could give you for skills into productions based on my experience. For the rest of it, you'll have to come to the to the talk tomorrow to learn how we're putting it into production on Superbase.

Any more questions? I'm going to be around throughout the whole event. So, if you catch me if you if you if you cross paths, feel free to to ask me anything. Tell me about what you're building. Love to see if it's with Superbase even more thrilled to hear about it. And for my end, once again, thank you very much. You've been lovely today for 9:00 a.m. Pretty cool good energy. So, just for my end, enjoy the the rest of the the conference and we'll see you around.

Thank you. >> Woo!