Full Walkthrough: Writing & Using Skills — Nick Nisi and Zack Proser transcript

This workshop is going to be skills at scale. We're super excited to be here with you today. There's going to be an interactive component and we also want you to feel free to interrupt us and ask questions as we go. >> Yeah, we'll show that slide again with the with the QR code and the the instructions to clone the repo. That repo has the skills the skills that we're working on plus the slides that we're presenting. So you'll have all of that as reference material.

I am Nick Nisi. I am a developer experience engineer at WorkOS. >> I'm Zach Prosser. I'm also a developer experience engineer at WorkOS and we're on the applied AI team. >> And this is like working with agents, that's just like what we do now. Zach, when is the last time you wrote a line of code by yourself? >> I think I did a CD in a directory recently. Otherwise it's been like probably six or eight months now. Maybe longer.

Yeah. >> Same. Same. We've been early on from the the Opus 3.5 days. >> From copying and pasting back and forth through GUIs to now. >> It's gotten a lot better and I don't know. There's a a mythos out there that it's going to get even better. We are WorkOS and if you are interested in like securing MCP for example or just setting up off for this new agentic startup that you're working on, reach out to us. There's a number of us here with these shirts on and we'd be happy to talk.

We're also hiring. So >> Thanks to the AI installer that Nick built as well. You no longer even have to configure and install off get yourself. It can just do it for you. So pretty easy to get started. >> For sure. All right. So as you know, when you're working with these with these systems, every single conversation that you have starts completely from zero. You're always just like passing in new information to it. you've got to reiterate how you do things and Claude never Claude for example never remembers that it ever talked to you.

It just continues on a conversation and so we have to provide that information fresh each time. Yep. >> Um so for example let's say you have a skill or let's say that you're talking to just in disparate terminal tabs. You're looking at different code bases over the course of a week. Every single time you start talking to it you need to reload all of that context first and say this is what I care about. This is how we do things here.

This is what we're particularly concerned with, right? It ends up eating a ton of time and slowing you down. >> Yep. And of course there's things like agent.md or Claude.md that you can put in information about the repo or about how you like to work like in a global directory so that it can remember that and understand it each time. You're giving that instruction each time kind of like appending it so that it will remember oh in this project we actually use Vtest and we use PNPM so you should use those each time.

So it can that's that's like the way to give it some memory for it to understand how to go. But it can still get it wrong. It can still decide not to follow things that you have. I've definitely had cases where I'm like do this this and then this and it skipped the step in the middle and I say why did you skip that? It was like oh yeah you told me to do it. I I didn't feel like it and that's that's how you know it's a real engineer. >> Yeah so one of the nice things about skills is that you can think of it as like a discrete unit of work where you can encode everything that's super important to you.

Everything you don't want it to miss. Everything you don't want to repeat yourself. It's almost like carrying if you will the dry pattern into the agentic era in a way and not repeating yourself. So and as we'll see that becomes incredibly powerful regardless of if you're a solo developer working on your own startup with 12 agents or if you're on a traditional dev team with 12 team members. >> Yep. It doesn't know what you know so you have to be very specific and be thorough with what you want it to know because it's not always going to figure it out.

Sometimes it feels like magic cuz it just does, but a lot of times you have to put in the work to do that. And that's what what like those memory files are like claw.md and other memory files. It even has like claw for example has its own built-in memory where it kind of keeps track of things that it thinks are pertinent to the the way you work or the project that you're working on and it will save that off. >> Yep. And and you know, of course this works too in just a single project context with some of those files that Nick mentioned, but then again the problem is that you're still tied to that repo.

Uh you need your team members to remember to pull updates to that specific project skill if if they want the context. Um there's no necessarily built-in script execution. So how do you get how do you interleave a deterministic result when you're having a non-deterministic conversation with an LLM? And eventually that starts to get pretty gross. >> Yep. >> So. >> And the there's downsides to this uh specifically these memory files where if they're tied to the repos that you're working on or you have to put them globally so it affects everything.

Uh and you can't do things like uh give it like more smarts like execute this script. You can, but it's kind of not built-in so to say. Um but that's kind of like where the things that you put in there are not always like transferable or portable to other projects that you might be working on uh and so we need a better way to do that. And skills are that next step. >> Yep, indeed. >> Uh it's a way to make things more portable uh and you can use scripts to inject real data uh and you can make them composable uh so that they can be very small and very focused on exactly what you want them to do uh and that way they're they're very small like a very small footprint in your your context window, but also uh you can build them in such a way that they are only going to be applicable when you actually want to do whatever that skill is set up to do.

So you're not just bloating the context with everything from the start every time like you are with a claw.md. >> Yep. And so if you haven't seen or heard of a skill before just to level set really quick. It could be as simple as a a single static markdown file without any scripts at all. Um but let's look at the difference of what might happen. Let's imagine that we're roasting a repo or onboarding a new team member.

We want to make sure that, you know, they're kind of up to snuff on how things work here. Without a skill, if you're just talking to any generic agent with no specific context injected, you're going to get, "Okay, looks pretty good." It's going to be generic advice. It might find some low-hanging fruit. If you say instead as little as 30 lines of markdown specific to your use case and your conventions, your >> constraints. >> Uh you can start to get back very, very like hyper-specific feedback about "This is how we handle, you know, routing in this project.

These are, um you know, we we follow semantic commits or whatever. And we've got readme drift here and that's unacceptable, right?" So, it it can take as little as 30 lines of markdown or less. And so that's the one of the first things that makes skills incredibly powerful is that it's a very minimal investment on your part and it could be as simple as a small markdown file and it becomes a composable unit of work that you can share across codebases and your team. >> Yep.

And you're you're codifying exactly what you want it to do and you have freedom to express yourself in the exact way that you want it to do that. And there's a number of different ways and techniques that we'll talk about throughout this workshop. Uh but it's much better having those skills and knowing that they exist and and knowing not just like that you can use them, but also setting them up in a way that the LLM can decide to use them when it makes sense.

Uh and it's going to give you a single repeatable way of doing that thing in the way that you expect it. If I just tell it some generic thing like, "Uh look at this and tell me how how good of a repo it is." Depending on the the model that you're running, uh maybe the the the amount of thinking that you have turned on, uh etc. It might give you more or less information, but it's never going to be the same thing each time.

If you want it in a very specific format, you want this report in this exact way, that's what a skill is. You're teaching the the LLM how to do something in the way that you expect it to be done and then it will follow that much more closely. >> All right, let's take a look at how a typical skill might break down. So, again, it could be skill.md, so simple as a markdown file. At the top, you'll notice a front matter.

So, think about anywhere else that you've used like a YAML-based system, almost like headers, right? In in other languages or formats. But, you got a name, a description, and this description is is incredibly powerful and loaded. This is what the LLM is going to use at runtime to essentially do routing and determine if this skill is relevant to the task that you've assigned it. So, that's kind of how the AI finds and routes to your skill.

And then additionally, you can provide additional context and and then even scripts so that you again think of it as your option or your on-ramp for interleaving determinism with the non-deterministic LLM conversation. >> Yep. So, the the most important things here are exactly what Zach said, the the name and the description. It's a misnomer that skills are only a single markdown file. They of course can be, but they're more like a folder with a skill.md file in them.

And then they can have anything else in there as well. And we'll kind of talk about that, but they can have references to other other things that that they might want to know. They can have scripts that it should run, and they can have images, they can have all sorts of different things. And then use that in different ways. But, the most important piece of it from the start is the description, which we'll talk about. >> Let's also just talk about constraints.

One of the things that's kind of not intuitive is that it can be more powerful just to to provide a few constraints as opposed to overly being overly prescriptive in exactly how you want the task done. So, if you provide just three constraints and say, "Never be vague." or "When you cite code, it always has to have a specific line and a Git commit reference with it." then you'll get better performance than if you end up, you know, bloviating in the middle of a markdown file so it's like a novel.

This is actually a common failure mode when designing skills. >> So today we're going to put all of this together into a skill that we're going to build here in the the workshop and it's just called repo roast. We try to think of like a fun generic skill that would be applicable to anyone who is working in you know JavaScript or different languages but also like like >> If you if you're not really like if you have an idea of >> What a get repo is this is applicable to you.

So it's kind of transcends all of that and is generally something that's useful for for everyone but also kind of fun. We can kind of be more or less serious with it as we're we're putting that in so it's going to allow for a lot of creativity as we go. >> and feel free to also kind of use this as a as a place to inject the actual constraints or the requirements that you have at work that you're kind of struggling with or testing.

So we'll kind of get the baseline together and then you can start customizing from there and we'll have some time to share and discuss them later too. >> So this is that slide from the beginning. If you haven't yet please download this clone this repo and work in there. We've got kind of the basics of the skill. And what we're going to do is just kind of get it set up and you can make it your own. We've got some general guidelines and some tips to do but the fun is going to be that we have a room full of people and we can have a room full of different ways of analyzing this and we'll also share that.

In that repo there is a share.sh that you can run and it will just ask you for your name and then it will put that into a KB store and then I can pull it down quickly on my machine and then run it against some repos like on the screen so we can share these at the end of the day. >> Yeah it'll be kind of a fun way to experiment with different approaches to the same skill. >> So speaking of loading skills we should talk about how skills load.

We we are generally kind of talking like you'll hear Zach and I kind of always just like when we're talking we're saying Claude. Cuz we tend to use Claude. How many here use Claude as kind of their daily >> daily driver? Well, okay. >> That's a pretty much what we expect. It's like 91% uh market. And then everyone else like there's Cursor and I have been dabbling with Pi. Pi is amazing. Uh but also Anthropic won't let me use my I I it's unclear.

Can I use my subscription with it? I don't know. Uh maybe I'll find out today or this week. >> You can pay more for it. You can >> Yeah. >> pay more in credits for it. That's fine. >> For sure. Um but when you're using these so you like the the main thing and the reason that we're so excited about skills too is that they're generally applicable to all of the major models. So Codex supports them, Claude supports them, Cursor supports them.

Uh the uh desktop apps like like Claude desktop supports it. So even like if you're non-technical you can be working on skills and sharing skills and and using skills. >> What was the skill that you did last week with the recruiting team in desktop? >> Yeah, uh I was working with our recruiting team kind of uh they're at an onsite and I was zoomed in with them helping them build a skill that could like take like candidate information and format it in specific ways and understand um you know what they're looking for in different things and kind of build reports automatically.

Uh so it's things that they could do pretty simply but they um because of like the the beauty of Claude desktop and all of the connectors that it has like it could just reach into Slack and pull in information from there. It could reach into Notion and grab that information and then mix that in with like the recruiting software that they use and put it all together into a a single report that then they can share to build from there.

So it wasn't like this is the final report that we use for everything. This is a building block that then they can use to do different things in different places. >> Yeah. >> And so it was really powerful for that. >> And as soon as you gave them that skill then everyone on the team is running it in a uniform way. >> Yes. >> That's the the power of it too. >> For sure. And so where do those skills go? Uh well, the the most basic place is if you have a repo there's you can just put a dot Claude directory and then uh have a skills directory and then a folder which is the skill name and then a skill.md all caps in there just like this and that will be a skill that lives with that repo and so anyone who is using that repo it'll just automatically load that and understand how to use it.

You can also have that same .cloud directory in your home directory and put those skills there and then they're generally applicable to everywhere that you would be using Claude. Same thing there there's kind of more standardization for everyone else on .agents. I wish there was like agents.md and instead of claude.md and .agents instead of .cloud but maybe we'll get there one day. So you can put them in there and if you've ever used like the MPX skills tool from Vercel that is just kind of symlinking them all into all of these different directories and so the skills are generally applicable everywhere.

That's just a an easy way to load and install them which is why it's so popular. Yeah. But the the main dev loop with it is you edit the skill, save it, invoke it, see what output it is and then do that process all over again and test it. If you're using Claude as well Claude ships with a fantastic skill builder skill or skill creator skill and that is really good for critiquing your skill setting it up in a way that Claude would expect it to be and even evaluating it which we'll talk about.

All right, so we're going to start by letting you go ahead and work and build the foundation so you should have that repo and we just want to get started with it. So the main things that you want to do is you want to set up a proper description for it. Now remember this description is not for humans. The description is really more for the LLM so that it knows when it should use the skill automatically. And so you want to set that up in in some way.

We recommend in in some way where like it describes like oh we're going to roast this repo and like like the user wants to roast this repo and get a an analysis, a fun analysis of it, or something. Be creative and and fun with it. But then you should just be able to like open up Claude and say roast this repo. Roast my repo and it goes and does it. >> Yep. And then remember that in general it's recommended that instead of being overly prescriptive in how to do something, provide your constraints instead.

So say we're using this format in this repo, or we follow these coding conventions, or we never do X or Y, and then allow the LLM to make the right determination at runtime. >> Yep. Yeah, definitely like closing it off like that. Don't prescribe what it should do. Kind of give it advice on what it shouldn't do and let it be more creative on things. But you can also like change that as well and be more assertive on things that you know you want in a specific way.

So let's work on that. Um a couple of things that like tips that we want to talk about in this first section is and this I think might be pretty applicable only applicable to Claude right now. I did ask Pi if it could do it and Pi just like made an extension that made it work. So uh that's that's awesome. But if you use the bang and then backtick backticks for like a script call, uh Claude will do like an interpolation of that.

Just like how JavaScript has like the dollar sign open curly brace and close curly brace. It'll just like instead of having whatever was in there like this, um where it's saying stale to-dos and then it gives you a command to run. It will just replace this with a list of the stale to-dos because it will actually execute this grep command and then do all of these pipes to all sorts of different things. That's totally not slurping up keys or anything. >> Yeah. >> Um >> But you you can imagine how this is really powerful if you're like say you're doing your morning report, your your kind of like your get status report.

Any of the pieces that you want to be output in a deterministic way, that's an ideal use case for this kind of script interpolation. >> Yeah. This is really great because you're not you're not saying go grab the latest commits or the latest 10 commits and give me some information on it. You're saying here are the latest 10 commits in the exact format that I expect you to understand them in, go and do something with that information.

So, it's not guessing, it's not going to be non-deterministic each time. It's going to start from this deterministic base and then go from there. >> It's also very token saving if you've ever said, "Go and figure out the 10 commits." and you've run that more than once or on three different terminal tabs, you know the first two might get it perfectly right and the third is like spinning and reading get docs and you know, before it finally gets there.

So, this is a way to say, "Once you've formalized a piece of your workflow, you can just codify it and say, 'Run this exact script.'" >> Yep. Yeah, like we said, without scripts, the AI is just speculating on what you mean when you say, "Go get the latest commits." >> Um yeah, and just remember that descriptions are routing rules, right? They're they're less for us and they're more for the AI to determine when to use it.

So, for a good example is you might have a couple different image generation skills and they're all kind of littered in the projects and maybe in your global skills. Maybe one is more applicable to your personal blog and you say, "On my personal blog, I always ship pixel art. So, if we're writing on this domain, this is the skill to use, right?" "If we're going to work, it has to be formalized and we use a completely separate image generation system or we only fetch images from S3." That's where you can kind of codify that in your description.

And if you're not sure, by the way, you can always ask Claude. That's the other like kind of secret hack of this era that everyone forgets is that a lot of times the models are capable enough that you can ask them, "Have I done this right or when would this apply?" Uh so, you can say as a test run, "When would you load this currently? If I only want it to run in these conditions, is this the best description for me or not?" >> Yep.

And a great example of this is when we were building this, I asked Claude, I was like, "Hey, I know I can do this, but do you actually support like skills calling skills?" And it was like, "Oh, let me go check." and it loaded like a Claude code analyzer skill to get that information and then do that research and come back and say, "Maybe." >> Yeah. It was like, "Kind of, but you probably don't want to do that." So. >> Uh so, your turn.

What we're We're to take some time to go do that. >> Yeah. >> Uh, till till like you go do that. Um, and when we do these breaks, too, this is a great time if you have any questions or have discussion topics that you'd like us to talk about, we can do that. We're trying to like fill the dead air of like you working on these with general topics. So, if there's something that you want us >> going to say that part out loud. >> Oh, that's okay. >> Okay. >> Uh, if if there's something that you want us to talk about, uh, we can definitely do that.

Otherwise, we've got some discussion topics that we thought we could talk about. >> Yep. >> Uh, but if you also if you have any questions or any of that, we can definitely >> I'll run over to the bring you the mic and feel free to shout out any questions. >> Yep. >> Um, but yeah, if otherwise then feel free to uh just start on this. And if there's any questions, let us know. >> Yeah, question. You want to run them? >> Sorry.

Where's the question? >> Um, you talked a bit about this in the beginning. But I always wonder where to draw the line between encoding instructions in like rules, cloud.md, and so on, and creating a skill for something. So, I'm curious if you have like what's your mental model to making that decision? Like have you landed anywhere? Like do you always start with the rule and then you make a skill if you can make it specific enough?

Or you do always start with a skill? Like how do you go about it? >> Yeah, great question. >> Great question. >> Uh, I usually Wait, like the the one the number one rule that you have to remember is that the skills sorry, the cloud.md or the agents.md, that is going to be loaded every time. When you kick off Claude, that's going to fill your context window. And if it's filling it with a bunch of nonsense that isn't actually applicable to what you're specifically doing, then you probably don't want it in there.

Um, I can show an example of like my uh what is it? dot Claude Uh, and then I think Claude.md if I can spell. This is my Claude.md. It's extremely small. Uh it just tells it that I want things to be a little bit more terse. Don't blow the eight. I just want to know exactly what you're saying. Uh be extremely concise. And then I also like I have this plugin that I'm working on. It's a a skill actually called ideation.

And I in here I put like some configuration for that so that all of the projects I I basically want them all to put like the ideation the artifacts that it's generating into my Obsidian vault. So it puts it all in there so I can more easily like find the connections between things. Um but otherwise it's like extremely tiny. And so that that's one thing that goes into it. If it's only relevant to the repo like like specifically you know I'm tired of it using npm when I wanted to use pnpm for example.

I'll put that in there like a single line that just says we use pnpm here. Um and then anything else like if it's you know more specific about testing or anything like that I kind of leave that to skills so that it's only going to be loaded when I'm actually like writing tests. >> To to the second part of your question as well the and we'll talk about this a little bit later but the other thing that's really fun to do is basically wait a week while working on it and then go back and ask Claude analyze my week's worth of work.

And then what are the skills I should split out of that based on this? >> Yeah. >> Um so again ask the system to kind of help you do that. Another question back here. Yes sir. >> Okay. You can >> Hear me right? Yeah? Yeah. >> Um so stop me if you're going to talk if you're planning to talking about this later. I was wondering about uh global skills uh which we will share amongst colleagues. So we're all at the moment with we've got I think 60 engineers.

People are writing their own skills. We're chatting on Slack. Oh I've got this great skill. It's really good. So then obviously uh engineering managers are like well we should be sharing these. Where is it where do we keep these? Where do we keep them in a repo? Um what's our artifacts library? And then others have said, "No, we don't want that because if I put my skill up and then someone's like, 'Oh, I'm going to change that.' Then we're going to have MR requests and then we're going to have to review changes to skills." >> Yeah. >> So then we'll get someone else saying, "Well, I kind of like a skill, but I'm now going to push my version of that skill with a very similar description to the shared repo, which everyone's going to get." And then suddenly we've got 10 from 10 door UX skills. >> Yeah. >> And they're all the agents then, "Which of these do I actually use?" >> Yeah. >> And we're wondering if you've guys have got to that stage of how to maintain.

And then the next one is 3 months later a new model comes out and these skills are actually a little bit too verbose. >> Mhm. >> So who's evaluating the skills and checking them and saying, "Okay, let's cut these from the global because now You get what I'm saying? This is where we're at with >> Yeah. >> how And so a lot of engineers are just like, "No. No skills. Everyone does it on their own. We are not sharing anything." So that was a bit of a rant.

You get where I'm coming from, yeah? >> Fantastic question. I'll take the first stab. Interested to hear what Nick says. We have published Maybe you want to pull it up like GitHub Work OS skills. That was one of the first places that we started publishing generic skills. And that's been incredibly useful because for example, I was building generic rag pipelines and then we found that agentic tool calling's higher performance.

So I can side load those skills that Nick put in there that are specific to certain documents. To the problem of individual engineers like saying, "I want a slightly modified version of this." I I I would almost say like in that case, cool. You've got a fork skill you keep locally. Um, and then to your question about, you know, evaluating the skill, I think asking Claude like with your current model, look at the skill using the skill builder.

Is it right for truncation or is there like, you know, additional extensions that we need now? Um, but I'll also share that we are feeling that same pain as I'm sure everyone else is and I think the management layer is just shifting to that kind of >> But they think even if you ask >> Yeah. No, you're good. >> Even if you then ask Claude, let's say a week later, a month later, "Hey, review our skills." There's 30 skills to review, and it comes up with lots of suggestions.

You then got to open a merge request for possibly one human or two humans, or maybe you we can automatically say the the person who wrote this skill originally has to be one of the reviewers. They have Have you got down to that yet, where >> We we I don't think we've gotten to that level with ours because ours started kind of formalizing documentation into buckets that were then easily side-loadable in different systems.

Um that does sound painful. I'm curious to think What What do you think about that, Nick? >> even got there. We're just People have just like foreseen that this is going to happen, so they're actually blocking us using shared skills at work. >> Right. >> Because they think this is going to be the problem. >> Yeah. >> Literally. We're overthinking it massively. We should just do it and try, but still interested to hear what you guys >> Definitely.

And I also think it's going to evolve rapidly, too, right? As we're seeing like they're still We haven't quite hit the LLM training wall, right? So, there's going to be kind of additional capabilities coming online, and and yeah, what does it look like in 6 months? Could we pair the skills down even further and get the same or better performance? Yeah, but yeah, I'll say that that's um Yeah, that sounds like a typical human problem of uh my skill, your skill, right? >> Yep.

We have a number of like to to build on that, uh we have a number of ways that we solved that. Like Zach said this um the skills repo, this is like our public skills uh that you can just install with like MPX uh skills add, uh and and those are all available. But then we also have uh some like internal skills that are more uh generally applicable to like engineers that work OS, and so it's like there's an auth specialist, there's a DX specialist, there's uh a ghost writer, different ones like that.

Um and then I have my own plugin marketplace as well, where I put a number of skills that are applicable to me. Uh and so I just load from all of these uh in different ways. We also have like a a big mono repo that like most of the engineers work in and you can uh a lot of skills just end up in there if they're mono repo specific. >> Yeah. >> Uh that's a much easier place, but yeah, it's the same problem like you got to get a review on it or it's got to be it feels kind of dirty cuz you're just like appending that to the work that you're also doing.

So, it's like an an add-on which doesn't feel super great on the PR. >> If I reverse engineer to some degree the plugin system, I think that's what they're trying to address kind of cuz you can also install like a version of a plugin the same way you can an NPM package, right? So, maybe that's kind of like the interface on top of the repo and then the tooling that I'm seeing everyone keep building repeatedly is like the tool that reads from a repo and installs skills into various places like tool easiness stuff that make that kind of like nicer.

But, that might be a solution to some degree where it's like cool, there is this mass you know, master skill of this, but I'm running this version because I need this fork. Um and then but it's not as gross as it sounds because there's an actual standardized API with the plugin interface. >> And it's all versioned. >> Yeah. >> No, no, that's it's great. >> When you MPX, do you then have flags? Do you then flag like I'm MPX public flag just front end or just UX or just product?

This just comes to my mind. I've never thought of Have you done it? Is it Is it >> I haven't, no. Um I haven't used MPX for that. I just used like the But like I said, we're mostly Cloud. I I've used the the Cloud Marketplace like the {slash} plugin Marketplace add and as long as your Cloud instance can uh access like an internal Git repo, it can just pull from there. Uh and so, that's what it does. >> It will it will pull all the skills even the ones that you don't need cuz you're front end, you don't want the back end skills. >> Oh, yeah. >> Yeah.

Then that that almost sounds like a packaging thing to me, but I did I did I think that you're kind of like in good company in a sense that it seems like you know, we're we're kind of got three marketplaces that are super relevant separate from or in addition to the project specific like skills, right? And then it just kind of becomes a matter of taste of each individual engineer saying like oh, I'm going to run this version of that skill.

But then something like the plug-in like interface is the way that you have a uniform way to approach it, which you could actually write docs against for onboarding and say plug-in add these three marketplaces when you come on board. And then if you're on a front-end team like plug-in install from the front-end marketplace or whatever the case may be. But that's like still at the end of the day on the back end that's like repo management.

That's right? It's similar to how it works with code. >> Yeah. >> Yeah. Great questions. Um yes, sir. Sorry. >> I forgot what I was going to ask. Yeah. Uh so it's actually two questions. The first one is do you do any like formal skill evaluations like a skill benchmark so that as you model drops, which skills are relevant? >> Yes. Um in the the public skills specifically on the the ones that I use internally, I am a little less formal about it.

Uh but the ones that we actually ship, uh we do ship uh in the Where is it? We have like a whole evals framework uh that we wrote to make sure that it it lives up to the standards that we have. And we're going to we're going to talk about this a little bit, but like it's mostly uh like doing several runs where it will load Claude without the skill and ask it to do a task and then load it with a skill and then it kind of has like a rubric on confidence or or like a a grade that it gives it and it's it it'll fail if that grade is less with the skill than it was without.

Uh it also fail like you know, it it tries to be I think 80% above or higher. So like 80% of the time uh or maybe it's 90%. Uh it's going to get this right with the skill and sometimes it gets it right without the skill. So the skill is maybe only adding 1 or 2% to it, but that's something that we track and keep on top of as new models drop. >> Yeah. >> Okay. Yeah. >> It's it's sort of fuzzy math, but it's almost like by having this this established baseline, you can at least test that way. >> Yeah. >> Uh it makes sense.

And then the second question was uh about um Sorry, one second. Right. Uh skill pick up. Uh so, if you get lots of skills, uh the models might ignore a skill or decide I don't need a skill. I'll just I'll just do it. What's your kind of experience with this to A like tested, find it, and then maybe improve it? >> Yeah. Um great question. We That that is a problem, and the more skills that you get like you can have conflicting skills.

Uh and so like which one is it going to pick? Um the solution to that like like for the Work OS one specifically like we try and keep it like for these public ones we try and keep it like very generic like mention all of the you know the the acronyms and things that we would want we would expect to cover uh from that. So, that'll trigger it to load. Uh and it usually does a pretty good job. You can also like if you're in a skill uh or sorry, in Claude, uh you can just do like Work OS for example like the slash command uh if you know that you want to do it.

And so like a lot of times we'll just like suggest, you know, if if that's what you want, I'll say like just run slash Work OS and it'll it'll load it. >> I'll I'll I'll call a skill by name if I want a specific like image gen or something. I'll say or I'll say like use the superpower brainstorm skill in order to determine a better plan. >> Yeah. Yeah. But also if you got if they really wasn't behaving, that's why you use the bang and then put a command. >> Right. >> Yeah. >> Um I had a question on um how do you decide when to create a sub agent versus a skill?

And can you reuse a skill into a sub agent? And there's just sometimes that um I'm going to create a skill and then I'm going to like uh maybe I should have written um a CLI cuz why did I even make a skill in the first place? And I I struggled between these three things. >> That's a great question. On the can skills can sub agents use skills? I actually I'm like blanking on that. So I'm asking Claude. And you can see that it loaded the Claude code guide skill to go check that.

So this is a great example of doing that and we'll get the answer here in a moment. But that's a great question. Sub agents is something that we don't cover a ton in this workshop, but it is something that's super valuable. And the the number one thing that I think is think of like when I think of what when to run a sub agent versus a skill is do I want it to have its own standalone context so that it can go do like a bunch of work on on something and then that's not eating the context window of like the main task job that that we're doing.

And then that way it can just like do a check in on that. And so for example, I have this ideation plugin. It's kind of like a a planner or a superpowers type thing that I like doing. And as part of that like I'm really like focused in on feedback loops to itself so that it doesn't have to bother me all the time about hey, you know, does this look correct? Or like tell me oh, it's done and it's totally not done. Like I want it to prove to me without me having to go look at the code that it's it did the work that I expected it to do.

And often times that's feeding the information that I would look for back into it and making it just go in a loop over and over. I hear there's a Ralph loops workshop after this. So you should check that out. But it's uh like in that case like when it's doing those reviews, those can like muddy it up and so like I kick off a sub agent to go do those reviews and then it just reports back like there I found these problems.

And then it just has a list of those problems and then it can feed back to itself to do it again. So I'm not eating that full context window every time. Yeah. >> Now also further confused by agent teams, which are different than some agents, too, right? >> Uh yeah. Another question. Thank you. And >> Yeah. I've got another another question here. >> Yes. Um I have a question about the the overrides in a skill. So, for instance, you you put a default and you say or whatever the user decides, but I find it's very random or it's I cannot really reproduce that and sometimes the overrides doesn't work.

Do you have any idea or like I I just want to find out what's going on. >> Uh my my best suggestion for that is just ask Claude. Why did you pick that over the other thing? Uh and how can I improve that in some other way? Like like you consistently or like you consistently enough pick the wrong choice or you don't respect my override. Why is that? What can I do to improve it? Um I wish I had a more clever answer, but usually it's just like I ask >> Yeah.

Just just ask Claude. He's good enough. Thank you. >> Great question. Question? >> Uh you called out uh superpowers was a skills library that you referred to. Is there other uh skills libraries beyond you guys that you currently use? >> Yes. Uh definitely. So, superpowers is one that I actually didn't use until yesterday when Zach showed me it. Uh and I I installed it. It has a number of different skills in it that are are pretty helpful.

Um these slides are actually written in slide dev uh and you might notice Well, you won't I don't know if I committed it. Um let me go to the full repo here. But in here, there's an agents directory and a skills and a slide dev skill and Claude might have had a hand in writing these slides. Uh which is really cool. We'll kind of talk about that, but like, some of the real superpowers I think are like when you assign it to do non-coding things, uh, because you really feel this magic.

Uh, and we'll we'll kind of show a demo of the remotion skill. That one blew my mind. Uh, it's it created a video based off of a prompt. And >> I I now use that as my So, every Friday when we have the all hands, and it's like what quick demo of what you got done this week, right? It just reads my get history for the week, and then builds a movie about it. Which everyone is was tired of on week three, and they're not going They're not going to stop.

They're just going to get more like I'm going to introduce characters, and it's going to get awful. But, uh, yeah, the remotion's incredible. It'll even pop up a like Chrome-based web editor where you can go and be like, "Oh, trim and cut, and let me add some fades, right?" Um, so that's insane. And then my favorite one that I probably got the most leverage of, uh, since installing it was just a I built it with Claude just a simple Python wrapper around Nano Banana. >> Mhm. >> Um, the image gen model from Google, which continues to improve.

So, I just say, "Hey, now it's on V3, go update it." And we'll show a little something later, but, um, essentially with that So, most images now that I generate with that, it takes like sub 7 seconds in a single prompt. But, using that same model, I was able to say, uh, take a single string from the user that's a prompt, say like a child running through a field. First, it makes that image. Then it uses their video API VEO, hands that static image to it, and says, "Animate this static image in the most obvious way possible." So, one user prompt of child running through field, nothing exists, and then 30 seconds later you have a video of it running through.

And I was able to to use that same method to do all of the interstitial scenes that I needed in a 32-minute film. And I am not a video person. Like, I mean, I like using like DaVinci Resolve and editing stuff, but I'm not an animator, and I was able to get all of that done in like may- maybe an hour. Um, so th- those are those are pretty trippy, too. Um, if you want to get really really down there like I've got I have like Claude reading my biometrics and stuff and like pushing back on me and telling me to like take it easy this afternoon cuz you didn't get any sleep.

So, um, but there's there's not like necessarily a skill for that yet. I I think the the ones that are really powerful are when Claude, uh, the other day blew my mind by saying it was also superpowers, "This is easier for me to show you the variants if I just mock them up in a web browser. Would you prefer that?" And I said, "Yes, please." And then it showed me all those and I was like, "A, go." And then we just built from there like saving countless tokens on just text like iteration. >> And I'm using that in a banana skill right now, but, uh, I just ran {slash} plugin and I'm looking at the marketplaces I have installed.

And some of the most important ones to me I think are the the Claude Claude Claude plugin's official one. Uh, I think that's where it has a nice, um, skill reviewer skill or skill creator skill, uh, which is really good. Uh, Obsidian is something that I use all the time. Uh, and so having the Obsidian skills and it knowing just how to use that. Uh, so it's based on what I want, but then also one that, uh, is actually very good.

Uh, where did it go? Plugin. Uh, is the um, Codex skills. Marketplaces. I don't know why it's not showing it. There, the OpenAI Codex ones. Um, it's not scrolling down. But anyway, that is like, uh, Claude does all this work. Codex is pretty good at reviewing it. So, this is a skill from OpenAI that just like pipes that to Codex and says, "Review this." And it goes and reviews it and then delivers that back to Claude.

And I have cut myself out of the copy and paste game of Claude said this and Codex said that and like going back and forth in in tmux splits. So, I'm I'm super happy about that. >> I'd say Vercel has pretty skills forward. They've got a bunch of CLIs and stuff that are are pretty interesting. So, they and they were like kind of the first on using some of the marketplace stuff, so uh, check out their like open source skill stuff too as well.

Great question. Thank you. Okay. >> That's what Anna Banana just made. Close. Close-ish. >> Yeah. >> Awesome. >> Uh but the fun thing about that is that you can ask it for any style. So, you can say like I I'm mostly do pixel art. Um so, I'll say like, you know, old school pixel art and uh it's a lot of fun. >> Yeah. All right. We are at time for this uh piece of it. Did anyone uh build a um a gen- a first like pass on the repo roast skill that they want to share?

Yeah? >> Cool. >> Awesome. If you want to run um dot slash share.sh. Oh, cool. Sweet. >> This guy, all right, wins the workshop. >> All right. Uh >> Are you Sharif? >> You're Sharif? Okay. Are you uh Okay. I'll run uh well, I'll just run off the table real quick. Uh so, I'll run them on the uh skill or sorry, this workshop. Oh, what did I do? >> It's pretty safe. Don't you want to run it against like work OS or something or the CLI? >> Uh yeah, I just realized that it loaded it locally into this one not in a global way. >> Oh. >> Uh I can do that.

Okay. >> Sorry. >> I'm going to give it the our work OS CLI and then I'll say uh repo roast Zack B on the uh CLI repo. We'll see what it does. >> It's a new verb, defending the hero bird. >> Oh, yeah. You can cu- you can uh customize those. So, I a lot of my uh spinner verbs are Lord of the Rings or The Office themed. So you'll see like defending as Giliath and things like that. I didn't I didn't think that would work, but that's okay.

It's running against this the workshop repo. >> Okay, that's fine. >> So it's running all of the commands that you gave and while you're while we're doing that, I will bring that up. So this is Zach's skill. Nice good description. Analyze repository health by running get and file system scripts to find stale to-do's, churn hotspots. Yeah, that's good. And then it tells it specifically how to find stale to-do's, awesome, hotspot files, largest files.

Nice. Constraints. Never be vague for evidence. Never present a finding without a script output or get data backing it. That's probably why it's running still. Oh, nice. Um yeah. Scope. >> Okay, Zachby. Okay. >> Nice. >> Nice skill. >> You didn't tell it to to just like be mean to you. So that's >> Be super mean to Nick and Zach while they're on stage. >> Awesome. >> Hi Amy. You look pretty mean sometimes. >> All right.

Awesome. We will we'll run more of these. We've got more uh more things to get through and we'll we'll do this again and we'll we'll test another one. >> Yep. >> Uh so moving on to the next section, we're going to make that skill smarter. So the first thing that you can do to make your skill smart is by providing more information to it. But this gets into the problem of the claw.md where you can be extremely verbose in there and give it so much information about your repo and you're just bloating the context window cuz it doesn't really matter.

Well, you can do the same thing in skills, but you can do it in a better way. And that is specifically with progressive disclosure. And I guess you could do this in a claw.md as well. But all it is is just saying like, "Hey, if you're thinking about doing testing for Excuse me, for example, and read through it." And you just give it like a path like a local path to testing.md. Or whatever. And that way it's only going to load that if it's actually doing like a testing skill or testing task as part of the skill run.

If it's not, it'll skip that. And so you can specifically tell it like, "Oh, in in this example, if you're doing like a scoring like if you're if this is a run where it's doing scoring, run the scoring launch load the scoring rubric and read through that so we explain to you how to score things properly. If we're not doing scoring, you don't have to load that and we don't have to fill up the context window with all of that blow." >> Yep.

This also gets back to that gentleman's question in the back too of like you can imagine this pattern really scaling out. So the way that it actually did scale out even in our public work OS skills repo you can go and check out. We have multiple migration guys that we publish for various folks. So like if you're coming from off zero, we'll happily help you move off off zero to get to work OS. And then there's N number of you know competitors essentially that we've got migration scripts for.

And so in this case you could say here's the migration skill and the migration skill is a pointer to the specific reference. So you're not bloating your context window. It's just loading the two markdown files it needs. >> Yep. And if you look at the work OS skill like it we literally call it skill router in there and it just has like a reference map. So if you're going to install off kit into next JS, you should probably load the work OS off kit next JS.md file from from And so if you're not working with next JS, we don't want to we don't want to load that and bloat that.

We only load it when you need it for all of those. And so, this file is just filled with uh routing to the actual pertinent information that you need. >> Okay. Um another way This is again, to some degree, it's fuzzy math under the hood, right? If you really get down to like matrix multiplication, but uh nevertheless, another way to boost performance here is to kind of force confidence scoring. And one of the reasons that Nick's ideation plugin, which is open source that you can go check out, works so well is that it has like an internal counter of confidence, of how close am I to fully fleshing out all of the variables that this task requires before I can go and execute.

And it then forces like a iterative loop with the user of continuously asking additional questions until it gets to the point where it's like, "I'm 95% or above confident. I've mapped most of this problem space in my head, and now we can start work." And the result as um you know, as a result, the the output is likely better. Um and so, you that same concept applies here when you're, you know, building skills, you can you can kind of add in that that same functionality and say, uh "For this particular aspect of the code base, you must always find this evidence and then get to a point until, you know, the tests are either this level of coverage or you have this level of confidence on on A, B, or C." Um and that's another way to essentially boost performance in the skills. >> Yeah.

Uh it's it's really important like like like Zach was saying, it's just like kind of pulling that number out of nowhere. If it if you say, "How confident are you?" and it's like, "I'm sure confident." Uh well, why? And as you give it like ask it to like show more of its work as to why it's confident, it might be like, "Oh wait, I'm not actually as confident as I thought." Uh and so, that's the whole thing is like trying to get it to think more.

Uh in the terms of of like the the ideation skill, uh what it's doing is it's using that to assess that it has like a full understanding of what I'm trying to say because like I have a problem where I don't give it enough information. I have the information. I know what I want. It's hard for me to express it to the the machine in a way that it expects. And so it's using that confidence score to say, I don't have like a full rounded understanding and it loads like a whole rubric on what it means to be confident on something.

But then if it's not confident, it uses Claude's built-in ask user question tool to ask me a number of questions to pull that information out of me rather than me being like, "Ah, you're not confident. Let me try and give you more insight." It's like, "No, I'm not confident because of these things." And then here's how you can make me more confident by answering these questions. And a lot of times it'll just give me multiple choice on like, you know, "Do you want this?

Is this what you mean? This is the recommended approach I would take, but if you want to go this other way, we could do it that way." And so like we have that dialogue going back and forth with it, but it's all based around how confident is it that it understands what I want and understands how to do what I want. >> Uh yeah, and so then this gets back to just kind of in practice the way this works or at least how it has for me and what we kind of recommend is, you know, build an initial skill.

Maybe you're doing that yourself in markdown. Maybe you're using the skill builder in Claude and saying I need a skill to do X. You're doing it you're using that skill for a couple of iterations, maybe a couple days, maybe a week. Um you look at what it produces and then, you know, keep in mind that as you're having multiple conversations with say Claude over the course of a week, all of those conversations are even getting saved locally to some degree in JSONL files.

And so you can um be honest with the evaluation phase about is this actually improving things? Is it not? Where does the skill fall short? What are the edge cases it's not currently capturing? What's the annoying thing that I've now discovered that I've been running it 7 days that it's missing? And then you kind of iterate. And but again, you're still going faster because you come back to a state that's already working and you say, "These three edge cases are driving me nuts and you also need to be be to like review your own PR's in the future, right?

And so then once that loop is is done, you have a skill that's significantly more powerful and then you can keep keep on running from there. But it's kind of like they're sort of evolving over time. Um so they're again like I think of them as like organizational units of where to put kind of, you know, work intelligence. And then over time if you're if you're doing it right, they're getting better. >> Yeah. When when skills first came out, uh Zach and I were actually at an onsite together in San Francisco.

And uh like we woke up one morning and they're like, "Oh, introducing skills." And we're like, "Well, this looks like every other markdown file that they provided. What what's the difference?" And um I like later that day we presented on on skills like I don't know, 4 hours later. And the the one that I built to present that was a Claude skill Claude skill uh that would analyze the like it wouldn't analyze your skill running cuz nobody had skills like 4 hours into them existing, but it would analyze, "Oh, you just did this task with Claude.

Let's go through and pick out what could have been what what we what we could like encapsulate into a skill so that it can do that in an easier way." And like since then there's like meta skills and things like that that have come out where it will analyze the performance of actually how you're using Claude or how you're using the skills in Claude. And then it can use that to feed back in just like Zach was saying, just by looking at those JSONL files, they're these logs of like the conversations that you're having with Claude.

And uh that can inform it on how to improve things. So for example, like in the repo roast, uh if it's kind of being wonky about how it's pulling in get information, adding in like the the bang with uh like the specific get command that you want it to run to get log information, that's a way to improve it so that it doesn't have to iterate over that and say and you go come back to it and say, "No, that's not what I wanted.

I wanted it like this." Like you you can be more explicit with it and that can be fleshed out by reviewing the performance that you had the first time or the first couple of times. Uh the The intuition I'll share is that um it's kind of like in my experience recently it's the types of nagging things that I find the most cognitive like resistance to doing every week that I actually need to turn into skills. And so like a breakthrough moment for me was realizing that like context switching between Slack and focusing on code and then going and ticketing like new asks in Linear was so disruptive to me that I just needed Claude to do that.

So now it just monitors and when someone asked me for something new in Slack it goes and looks at my Linear and then if there's not a ticket for it already it does de- deduplication, adds a new ticket, and then I'm haven't left my flow, right? I'm still able to focus. And so like that's kind of the intuition I have now is that um you can sort I think it's really powerful and I think we're only at the very beginning of it like analyzing your own workflow over time, feeding it more information about how you actually work and then letting LLMs, you know, do what they're really good at and compress down that that actual time. >> Is there a skill out there that you'd recommend for >> Uh is there >> Is is there a skill that you mentioned that there is skills out there that may have skills to review your kind of past conversations and propose skills or improvements?

Is there one that you use or would >> That one there's not one for that but and I didn't do this myself the last time. This was like last week but what I should have done is say, "Hey Claude, use skill builder yourself." Cuz Claude's got that baked in skill hyphen builder I think it is. Use skill builder to look back at my workflow and tell me where it's the least efficient. Right? And then that's also pulling in connectors cuz there's a Slack connector and there's a Linear connector.

So that's where like the markdown might be referencing you must always use the Slack connector to pull in this and I only care about these channels and direct mentions of my name, right? Um but yeah, I think it might even be faster in some cases to just say, "Here's where I work. This is the tool that we use to communicate. Make me a skill that does that." It's also like kind of crazy. >> I think this is the one that I was thinking of specifically is Claude Botiscal that helps you configure Claude including like setting up those skills.

I think this is the one I've used but like Zach said I've also just asked it to review its own performance and kind of go from there. One really great thing is like I built this pretty cool tool and I wanted to write a blog post about it. And it was all built with Claude so I was able to just go ask Claude hey remember that time we did this fun thing together? Let's reminisce about it and we just like talked about it and it like led to these anecdotes that I added to the blog post that I I completely forgot about but they're Claude was very fond of that moment between us. >> That's not what's happening.

You don't understand that. Okay. >> No, I don't. >> Under the hood that's not what's happening. >> Don't lie to me. >> Do you use any skills for memory for maintaining a like memory state within Claude? >> Great question. Claude has its own memory built in and I there's that auto dream thing. I don't know if that's real yet or if it's like a a thing that's coming but it will actually like prune the memory. And so I've been like focused on building around that but I've been building it on in Pi specifically and so like I built this I built like what it would take to be a DX engineer at WorkOS as like a full agent using Pi and it's called case and it uses memory internally like memory.md files and it works across all of our open source repos.

So it knows like React and React Router and Next and TanStack Start and all of those. And so then it has like general memory files and then like framework specific memory files and it goes in and prunes those and updates by doing like as part of its flow doing a retrospective at the end and analyzing its own performance and then saying oh I spun I spun in a circle a bit for this. I could have like once I got to there like I can just save that to memory so I know like this is the command I run next time to get the information I need.

And it just keeps track of that. I haven't built in like the full dreaming thing where it prunes that yet, but um I I'm experimenting with it. >> Yeah, and also I want to play with the Obsidian connector more because I think that would be super powerful. I I had a habit in the past of using Obsidian and just making a daily to-do with it just the date as the title. And then so I think writing to and reading from those vaults so that you could imagine saying, "Look back over the last week." Last week gets translated into what are those actual dates?

It fetches those files directly, right? And then it can also write consolidated memories. It's also worthwhile playing with things like Open Claw which I've I've done because that memory system that ships with was surprisingly good, better than a lot of like stock Clouder Open AI's stuff. And so seeing how it does that with like daily journal.md's and then consolidated memories which I think the dream stuff is kind of pointing towards like consolidating memory over time. >> Yep. >> Um but a lot of times the crazy thing about this is like the answer is one-turn request with Skill Builder is the fastest way. >> Point it to >> Yeah. >> This is a good repo to test it. >> Yep. 100%.

Yep. Um so we're going to jump into the next uh piece of of work on your side, and that's adding phases uh and confidence scoring to it. So adding progressive disclosure, uh adding a confidence score, telling you like how confident are you in in this? Or like like uh we we've got some examples uh of that potentially like uh you know what's a good example? >> Um How confident are you in this, you know, you installed off Kick correctly? >> Yeah, but I mean like for repo >> Oh, Rust.

Oh, for repo Rust? >> Uh you you know, you gave me a bad score on I don't know, Git commits. Why is that? Like >> Okay. >> have it dive down deeper than just >> Yes. Oh, this is our pattern of how we use Git commits. We always have our messages like this. We're following these conventions. So then based on that, what's your confidence that this is correct to our repo? >> So for example, you might use uh conventional commits at your work.

And if you find commits that aren't like that or you find a bunch of merge uh commits in there For shame. Uh but yeah, like different things like like that you can you can add specifics to and have that as be as a um a progressively disclosed rubric that it can follow for those things. >> A quick housekeeping thing in case for any reason you're behind or feel behind, you can run setup.sh and then checkpoint two to get to the same spot that we're at now. >> Yep.

Right. >> And then yeah, any other questions feel free to shout out and I'll run down there. >> Mike. >> We'll spend about 5 minutes here and then we'll move into the next section just to make sure we have enough time. Do you want to talk about um any of these topics, Zach? I can talk about when confidence when um yeah, confidence scoring saved us. >> Yeah, what's that? >> That was when we were working on the um when confidence scoring saved us.

The uh well, that was kind of built into the evals uh that we wrote. Like Claude ships with a whole eval framework now that you can use and it'll like spin up a GUI for you or like a it'll create an HTML report and you can see like before and after and all of this insight into how your skills are running and whether they're actually like improving Claude or making Claude worse at the task. Um but before that existed, uh I was writing my own to do that and uh it was all based on on that.

And so like let me let me bring up the um ideation skill. And I'll just say let's see. We'll go to the CLI. And I'll say >> So for context, this is our Workless CLI that we're building in that yeah. >> I'm on the main branch of that. I use work trees for that. Um what's a feature that we want to add? I want to add a fun {slash} buddy command similar to how Claude code shipped that for April Fool's Day. I used a a tool called WhisperFlow to go full Wally and not even type anymore.

Uh and I just press a button. This is how I code now. Um >> Do you prefer that over the Claude's voice mode? >> I Yeah, I do. I've been on WhisperFlow for maybe a year now and the thing I like about it is that it can uh input anywhere on on uh Mac. So, you know, if you're in uh some funky old like website in Chrome, it works there. It works in Safari. Works on any app that you've got as long as you can focus a cursor there.

You can insert text there. And it's also fine-tuned towards like technical terms. So, you can say at user Um you can reference files, etc. So, it's great. I mean I mean I imagine that more and more of the tools are going to get their own native voice uh over and over time that's going to become like a dominant like interface, but right now WhisperFlow is like a pretty sweet experience. Yeah. >> I turned on fast mode so it'll go faster.

Um yeah, it also does cool things like uh you can say like when you're dictating into Slack, uh be more casual. When you're dictating into an email, be more formal and it will kind of >> It's sort of context aware in the formatting that it'll put out like it can say it knows you're in Gmail or it knows that you're like writing code, you know. Or requesting code. >> So, this is an example of the ideation scale. I gave it that that simple command and now it's saying like, "Oh, what do you like I don't fully understand what you mean.

Uh what kind of fun are you looking for?" I'll say, uh a visual gag. Uh ASCII art gallery, sure. Hidden Easter egg. Uh yeah, we'll go listed but subtle. So like I I give it one sentence. And it's like, "Well, what do you mean by that?" And it's like pulling all of that out out of me. >> there's a value in thinking. It's like, you know, the same way that a good engineer in a whiteboarding session would kind of draw the same stuff out of you. >> Yep.

And so right there, it did this confidence score. It's based on the problem clarity, it has a 20. Goal definition 18. Success criteria, it doesn't really know what I'm asking for. So that's the lowest one. Uh scope boundaries and then consistency. So those all add up to 100 and I got a score of a 90 out of 100. So it doesn't It's not going to just be like, "Okay, I know what you want." It's going to ask me uh some more things.

Like, "Oh, we'll do that and we'll just have minimal up." I'll say zero config. I just want it to go fast. And so now I'm at 96 out of 100. So it understands what I want and now it's going to write a um a contract for me to read. I read and review the contract and then it's going to build from there these phases that I can execute or these specs that I can execute in phases uh and then go from there uh so that I can clear the context for each one and have like a fresh context going. >> Yeah, the way I would say that is like is the math here tight?

No. Uh does it matter? No, because the value is in the iterative loop of like clarifying and and clarifying your own thinking by by responding. >> Yeah. Oops. And so there's the contract that it's it's loading. Uh and it tells like what success criteria means, the scope boundaries, what's in scope, what's specifically out of scope, any future considerations, how we plan to execute it. This is an easy one, just a single phase.

Uh and so it's going to just create that spec for me, which it did here. And then I could run this uh and go. And so it was all gated on that that confidence score. >> Cool. >> All right. Um You want to jump into >> Yeah, let's do it. >> All right. Well, we're going to skip ahead into the next section and we'll have one more one more thing and we'll do some sharing after that one. So, kind of moving beyond the editor, we consider we we thought about this and we're like, does that title make sense?

Skills beyond the editor cuz we're not really in an editor, but like for us, we kind of are. Like we don't open I don't open NeoVim nearly as much as I used to. Uh so, I've lost my identity a little bit. But um yeah, these skills, they really do work in a lot of different places. Um another thing that you can do is like you can level up your skills in a number of different ways. Uh so, like for this repo roast, for example, you could have like, oh, I want to know who the bus factor people are.

So, use like a get short log to understand who's committing the most, who's committing the most in specific sections of the the code base. Uh and, you know, list out what the bus factor is and how vulnerable we are to that. Commit crimes. Uh this would be people who just have bad commit messages. It's so easy. You just tell Claude commit it and go. Uh zombie branches. You could have, you know, list out all of the branches that never went anywhere or that are still hanging around.

Uh who is committing at 3:00 a.m.? Who's who's up the latest uh working and making us all look bad? And then this one is definitely something that you should you should add and that's >> Is my read me >> Yeah. >> Is my read me real? Does it explain or describe real things? >> Uh yeah, and so again, the reason that this is so powerful is that it's no longer specific to any foundational model provider, right? Uh you can define these skills and then you can use them locally in Claude code, but you can share them with your team as we talked about with, you know, get based, you know, plugin architecture.

But now you can also put them in Claude desktop and web as we talked about with the recruiting team. Folks that identify as completely non-technical are loading uh specific skills and running them in their own sessions. >> And sharing them. >> And sharing them, right? And then now as we're finding like agent harnessing is harness is becoming more and more relevant and so things like Pi which is what Open Claw runs under the hood, you can load them there as well.

So it's it's the value is really in like defining the discrete work block and then figuring out exactly which tweaks make it the most effective description of getting that work done. And then, you know, sharing it with your friends and putting it on different boxes without having to do much more than authoring some markdown and possibly some scripts. >> Yep. And skills, if you took a skill file like you took repo roast with that skill.md and any scripts or references and all that but you took that folder repo roast and you zipped it, you'll get a .zip file back, right?

Rename that from .zip to .skill and now a non-technical teammate can drag that into Cloud Desktop and use that skill. And that's just how they're shared. That's a really easy way to to share them. Not a really easy way to version them. >> There's still there's still pain to around like how do you handle sensitive you know, credentials in that case like you you don't do it that way. Please don't put it in the zip file.

But, you know, it's evolving. So. >> But you can also use those marketplaces. Like the Cloud Marketplace works in Cloud Desktop as well. So that's an easy way to to share skills if they are applicable to like non-coding workflows. >> For sure. >> And so some of the like we've talked about this but like one of the things that I really wanted to drive home is like with the Work OS CLI, this is a it's like a generic CLI that you can use to do like Work OS commands in it but like its flagship feature is this ability to just run install.

So if you have a project that doesn't have auth in it or you have like other auth in there that's not Work OS, you can just run Work OS install in there and it's going to politely remove the other off that you might have in there uh and then add in based on what you are using like if you're using next JS or tan stack start or whatever it's going to figure that out and to load that in there for you and the CLI is using the cloud agent SDK which is like a program programmatic cloud code that you can ship I can ship in the CLI and the smarts of that all of the brains are actually skills that are in the work OS skills directory so it knows all about that and the reason we did that is so that we just had like the you know two birds with one stone we have we build the skill and we make it good and then we prove that it's good by having the the CLI run it and the beauty of the CLI is like it's an easy command you just do NPX work OS install and we're like proxying all of the commands to Claude so that it hits our API token and and so it's an easy way to just like say here's a zero friction way to get set up with it it'll even create like a work OS account for you and you can go back to it later and claim it so it's like five minutes and you're you're set up and all of that is entirely skills driven >> Yeah another place we're seeing like high leverage with this is imagine blog writing like lots of folks on the team as it's growing like want to write blog posts in a uniform way but they don't know exactly how our CMS works exactly the the tone or format and like the conventions that we use and that's the type of thing that you used to put in a notion doc and then hope that you could inject it in someone's slack and like force them to read it before they write something it's just easier to define that as a skill so that they can interact with it and then get to 80% of that artifact without having to consult somebody else essentially >> Mhm >> code review image generation with image generation too you can also put additional parameters there to get like specific styles as well CI pipelines and as I mentioned earlier in the talk like once Nick had published up the you know public repo of work OS skills the rag pipeline was able to just start loading them all as agentic tool calls and performance on all those queries just jumped over just, you know, flatly chunking all documents and putting them in a vector database, for example. >> And you saw the giant lobster outside when you came in, right?

Like that's all can be skills based as well. So it's skills are just this uniform way that transcends the the cloud code or the codex and it's something that you can load anywhere at any technical skill level. >> Yep. >> So it's really easy. Uh we talked about evals like measuring this stuff matters. With the the skills like specifically with the next JS installer skill, I actually found out through my evals that I was making things worse because I was overly prescribing what to do with next JS and cloud code was just inherently good at working with next JS and I was making it worse by being too dogmatic about what I wanted it to do and it led to like a 30% drop I think in like overall accuracy based on these numbers I made up.

Uh but I was able to use the that and I I kind of think of evals a lot of in a lot of ways like my Apple Watch. It tells me like my heart rate and you know, how how many calories I'm burning throughout the day. Is it accurate? No, of course not. But it gives me a general like baseline of like I am more active today than I was yesterday and I can kind of use that to gauge where I go forward. Is it accurate in what I like base my my life on it?

No, but it's a a general like vector that I can I can look at and see whether I'm improving or making things worse. >> Yep. >> So some skills in the wild, Zach you've you've made a couple of skills that are these are specifically like not code related but they they're pretty impressive. >> Yeah, so this is what I was talking about earlier just to show what I am the most excited about is like taking what seems like an incredibly complex workflow and then just making it available as a skill.

So this is as this is an example where I have a a Slack avatar that I built I had generated for me like months ago and I just handed it to this animation skill and I said animate this in the most obvious way possible. We'll see if that's actually obvious. So taking a giant ball of energy and grimacing at it as one does. Um but the point is that was a single text prompt. Uh of like make this person look like they're in Fallout holding a ball of energy and then animating it.

This one uh is the same exact skill. So the same markdown file and two Python scripts saying, you know, the prompt was child running through a field. And there's also sound with this because it's um hitting the VO API. So again, the it first uh you know, Claude reads the markdown skill, says, "Okay, I understand what this is. It's a a sequence of two API calls I'm going to make. The first API call is the user's prompt to make the static image.

The second API call is the output of that, the static image, and then a new prompt that I write saying, "Animate this in the most obvious way possible." hitting VO with uh VO API with that and then getting back an animation, but you know, again, that's like 30 30 seconds of generation time. And so I use this exact same workflow to to do like all of the interstitial scenes in a in a film recently. >> Uh and another example, uh I mentioned this earlier, but the remotion skill.

Uh I have I'm terrible at video editing. I don't know anything about it. Uh but when I was working on the work OS CLI, I thought, "Oh, it'd be kind of cool to make like a fun video that I could use on Twitter to like demonstrate it or or talk about it." And so uh somebody was mentioning remotion and I just asked it to make this. And it put it together pretty much like this. Like I asked it to use our our actual logo rather than some made-up one.

Uh but it even like understood like the output of of the CLI and put all of this together into a demoable video uh that showcases what it can do. And I didn't have to do that Uh at all and I I looked super impressive without knowing anything about video. And also like the skill when I said do this, it loaded up a like localhost 3000 in my browser that was a full remotion video editor. And so I could see it playing on a loop in there and it was like doing things and I'm like oh you didn't use our actual logo.

Go use that. And I just like told Claude to do that and it just updated like in real time. It was it was so cool. >> Yeah. So imagine like hooking this into your GitHub CICD flow and then at the end of a big project or every time a milestone gets merged, you auto update, you know, whatever document and then even include a demo. Um it it can start to get pretty powerful if you orchestrate skills that are well defined. >> Is this the skill? >> is exactly how that that one works under the hood.

So you can imagine like the one that I showed you that had the two YouTube videos. So if it's called animated image, the first one's going to be gener- generate a minimalist static image and then um take that image and animate it via video. And you there's just two scripts. There's one to generate an image here and then there's one to generate the video. Um but the skill file itself is like 30 lines of markdown. >> Yep.

And that that nano banana one that I ran earlier, that was just like coming up with a creative enough prompt, like taking the idea that I had, like flushing out the prompt, and then it passed it to a TypeScript file that called the nano banana API and got the image back. So uh that skill is just basically like a a simple LLM wrapper around this uh around a TypeScript script that uses their API to to go do that. >> So it's also like just broadly applicable to workflows.

It's not just a dev thing, right? You can imagine if sales has a very specific way they have to reach out to people or there's always like a type of report that you're generating for customers or prospects or whatever. Um all all of this is like excellent for use with skills. >> So did anyone have a a skill a Replit skill that they want to share? Yeah. Okay. >> Yeah, the Amy and Wolf from Raven Wolf skill try that. You have to see the results. >> Which one is it? >> A number two.

I uploaded another one. Number six is the used one. Used. >> Okay. >> Used. >> Audio number six. All right, so while that's running No, we ran the wrong one. There we go. While that's running, let's go look at it. Oh, nice. Okay. >> Ruthless honesty. I love it. Brutally honest. With a heart of gold. Awesome. I love the context. Lots of uh >> Six files, very nice. Excellent. >> And here, uh the audience detection. You told it to load audience guide.md.

Here's that progressive disclosure about that. This also just helps to keep your markdown files manageable. >> This is a 10 out of 10 skill. >> Yeah. >> Very nice. >> This is awesome. All right, let's see if it gave us anything. So it's it's grading the the workshop itself. Um six out of 10. >> Feel that. I thought we had I thought we had something going, but >> Hopefully you give us a little bit more of a of a grade than that.

That's awesome. Uh some critical feedback. >> on fire cuz it doesn't exist. Yeah, that's great. >> 1,200 lines of monolith, yeah? Yeah. >> Get identity crisis Zach is two people. That's how it feels, too. >> Hard coded secret, that's okay. It's not really a secret. >> That's awesome. >> Love it. >> Super cool. All right, we got 3 minutes left. There's any questions or um anyone else want to share a skill? >> So, this is a skill that you can use, but more importantly, it's techniques that you can take and use to build your skills and build them up in different ways.

There's a lot more advanced topics that we can go to go into as well. We mentioned like sub agents for example. Sub agents is a great way to extend those skills without bloating the context and having it kind of do one-off things and then and then exiting. um And the like to take this to the next level, I really recommend like having Claude's own skill creator skill >> Yep. >> installed because you can just say, "Hey, I have this skill.

Is it any good?" And it'll give you pointers. Or you can say, "Run some evals on it." And it'll run like a full eval test suite on it and tell you, "Yeah, it's good." or "No, it's bad." >> Yeah. >> And and can go from there. And then like Zach was saying, like reflect on the transcripts, reflect on how you're actually using the skills and you can use that as insight to see how to improve the skills and the execution of those skills.

Like for example, if somebody kicks off a skill and it's always asking questions about you know, a specific thing, maybe that's something that you can provide ahead of time. Or if you see it like, "Oh, it's going and doing like 10 tool calls." Maybe you could like condense it down to one or two tool calls and pre-provide that information so it doesn't have to do that each time. >> Yep. The plus one recommendation on the internal skill creator and then the other thing it kind of suggest I mean suggest to you to do over time is to think about the way you manage your context, even stuff that you used to think of as disposable, right?

So, like in the pre-LLM era, we might have dev'd real hard at the keyboard all week, as I used to do, and then finally on Saturday, like wipe it all away so that I can GitHub for myself. Like now, all of that context is gold. Like the conversation, especially what failed, especially what didn't go well, especially what was frustrating, because now all of that is very rich context for a skill creator or refiner to mine, and then build you a bespoke tool that's going to solve that problem smoothly next week. >> Yeah. >> So. >> And you can also like think of skills like you could use that progressive disclosure to like disclose things to different audiences.

So, for example, you could, depending on who's running the skill, you could say like get config user.email and and figure out who this the user is. Or you could do things like, "Oh, how many commits does this user have in there? They have 10,000 commits in here. Okay, we can really roast with them." But this other person who has four commits, they're probably a new hire. Maybe go a little gentler on them. Don't scare them away from this project that they just started in week two. >> Question, yes. >> Very quick, Zach.

Could you take me through again you're saying about the context switching you'd somehow hooked up Claude with Slack and Linear? So, it sounded almost like it's constantly being able to read what Slack's doing. >> Absolutely. I have it um >> Claude co-work or Claude We use Cursor, so I don't know if we have the same. >> Got you. Yeah, I'm using Claude code now. It's possible to do it in Claude code and and Claude desktop, but essentially I just have the connector in Slack, so I say I had to do GitHub or just had to do OAuth with Slack, and then it can read my Slack messages.

You can now run the loop command, at least in Claude code, to have it like do that every 15 minutes if you want. And then you say in the prompt, "If there is not already a correlative Linear ticket, make a new one for me. If there is one, and there's additional asks on this, you know, request, update Linear." And then, by the way, you have a second terminal tab that's looping against your Linear state. Kathleen works at WorkOS.

Earmuffs, Kathleen. I'm really working really hard. And you have the second one that's looping and looking at your Linear tasks and then like doing work for you. Essentially. But the the main point was just that um yeah, we're sorry, we're out of time. The main point was just like automate those loops. So, that's our time. Thanks so much, guys. Thanks for being an awesome audience. Thanks for all the great questions.

Really appreciate it. >> Thank you.