You're Automating The Wrong Layer (How 30,000 People Build AI Without Frameworks) transcript

Interpretable context methodology. That's the name of my paper that I published about 3 months ago, and now thousands of people and hundreds of GitHub stars, as well as over 30,000 people in the community, are all using my methodology to build their own systems with AI. But they're not building multi-agentic frameworks and all these absurdities, they're building folders and markdown files on their computer and getting huge results from it.

Now, I have a quite a few videos on that, but today I wanted to talk less about and more about what I've been building, what me and my team have been building as a future thing. What can you actually build using this methodology? And what happens when you bring in some traditional software programming into it? Now, for those of you who are new to my channel, essentially my methodology is about structuring folders, structuring markdown files.

It's very similar to the LLM Wiki that Kaparthy, the founder of OpenAI and currently just moved to Anthropic to teach there, uses and is very big in Anthropic's actual use cases. They are very big around skills, which is the idea that the right amount of scripts, processes, and ideas can all be kind of broken down into just plain text and folders, and the agent can navigate that to get the context. Rather than using really complicated workflows or infrastructures and harnesses, you can create one really good harness for a single type of model, and then that model can then navigate structure as it needs to.

Instead of creating rag that pulls everything in, you just give it access to normal databases. You give it access to folder and context structure. If you haven't read anything or dive into this process, I do a deep one with Claude Code in my video right here that you should see tagged. Essentially, it just looks into how to use this, how I use it, and a lot of people are seeing 20, 30, 40% token reduction, and really just getting outcomes a lot faster without a lot of infrastructure.

Even further, you're able to use it to really build out teams and build out processes that traditionally would have taken, you know, a vendor 6 months and an entire infrastructure update versus just working with them. And as the people learn how to use the AI, they're building the structure for you. Your team become the vendor, and then you can hire technical talent or bring in your own technical talent to amplify it, look at the security and governance behind it.

But all of that aside, let me chat about transcripts, about dialogue, about content, and why it's so important to this entire process. Now, this is an actual slide that I gave to a real enterprise company that's bringing in, I think, 60 million a year or something like that, and they have learned and used this process. One of the biggest things that I try to share with people when we're trying to understand how to use AI, how to use these tools in your workflow, is my layer process, right?

Layer one is where you're just chatting with a chatbot, you're copy and pasting. The layer one is this concept where I started years ago when I first started messing around with language processing systems, and a lot of people are today. It's where you log into ChatGPT or Claude, and all you're doing really is just kind of pasting in ideas, right? You're just chatting back and forth in it. You're copying and pasting, you're throwing in concepts in there.

Um you're just asking it basically questions and things like that. What you actually want to really understand is the amount of effort to get something back is low, and you can get some pretty decent things, but the amount of output and impact that you get is actually very weak, right? It's not going to be huge. You can get some very good stuff, but earlier, 3 years ago, it was very challenging to get any sort of outputs.

And in order to really get a better output out of it, you have to have longer and longer conversations. As in, you're essentially adding multiple chats, you have saved prompts, maybe you have a tone style. And we see a lot of people sharing prompt libraries. They have shared prompts. Maybe they're adding the prompts in little by little. A lot of other like kind of integration tools do this where they're grabbing a prompt and giving it to the AI automatically.

It's doing chain of loop processing. And these skills are actually the layer two. Someone else went through that layer two workflow, found the right amount of prompts that you actually want to be able to give in what order you want to give them and kind of handled that as a skill. But then skills evolve even further and you realize you only need certain parts of a skill at a certain time. You don't want to overload the context window and you want to start creating automations.

That's where you start leaning into level three or layer three, which is where you're actually doing this huge workflow. You're taking multiple skills, multiple prompts, multiple workflow processes, maybe even multiple AI and actually linking them together. Then they realized that you want to do even more steps. You want these agents. And that's where you have these other markdown files or scripts in many areas, right?

People actually put Python for people to actually run for the AI to actually run itself to be able to handle the creation of stuff because they needed more determinism. They wanted the output to be the same every time. And so if they could package all of that into a prompt, all of this could be uploaded and sent in the right order, that's where skills come from. And this allows them to then in this case create more skills, but you have hundreds of skills available.

And if you have Claude installed on your computer, you can actually come through and look through these skills deeply. Some of these skills, right? For algorithmic art, a skill for creating art with like kind of an algorithmic process, has an HTML viewer and a template built into it. Maybe you're trying to create a doc file in markdown. You have the actual skill explanation. This is how you do it. These are the prompts they put in order.

Traditionally, 2 years ago, I was putting these prompts in manually. First, I would I knew that okay, if I told the chatbot this in a certain order, right? I come in and I paste this here. Right? It would do that process. I would wait for its output, verify that it's right, and then I would move on to the next step, which is okay, well then we need to edit that and have these things. And over time we realized we could drop that into the AI all at once and it would recognize that.

But then we could drop all of it into the AI at once and tell it to reference certain areas or actually use commands and the single agent could then do those things. Right? Now, traditionally, before, the way they solved that is using LangChain or Semantic Kernel or all of these things to do these actions separately or to inject these at the right time in the thinking process of the AI. But now we can just break down the thinking process into folders and structures and trigger them as they're needed, which is what these folder skills and processes are doing.

But again, a common theme. There's a reason I'm describing all of this. All of these skills, all of these folders and markdown files, all have one core theme. Discussion and dialogue. They all came from an initial layer one of discussion and dialogue breakdown. At layer one, you're pasting a prompt into ChatGPT and then copying the output or going through iterations of an output. At level two, you already expect what the output is from level one, so you have refined prompts and processes that are automatically being injected in.

At level three, you're doing all of the level two automated multiple times so that you can then come in and look at these processes. And that goes for code review, data reporting, writing and drafting, all of those. That's what we really want to understand here. But it all comes from dialogue. Conversation has the structure we're looking for. The intent is carried in the conversation. And instead of just describing this randomly, here's a tool K, who is one of my co-founders and a really brilliant person currently studying at the University of Edinburgh, created to really help exemplify this process.

So, imagine here on the left, this is the chat. And we're going to break this down to the simplest process. This is a chat that may have happened forever ago. And the idea is, can you tighten this paragraph? It feels too wordy. Climate change blah blah and you put in the paragraph in there. Now, in the past, right, the agent would just say, "Here's a new one." And then we would say, "Hey, this feels weirdly formal now.

The original had more of a conversational rhythm." And so the agent does something and then it's this constant back and forth until we finally get the output we wanted out of it. Now, everywhere along this conversation, even in this simple one, there is a chain of decisions being made. And if we actually extract those chain of decisions using K's tool here, here on the left is the decisions made by the user. Here on the right are the decisions made by the model.

Now, when we look in, the goal was to tighten a paragraph. All of this is all of the goals and focus. And as you can see, it links directly here. Can you tighten this paragraph? It feels too wordy. Now, we set up constraints along the process and at different points. Perhaps there was a constraint saying, "Hey, there was too much wordiness. We want to perceive the rhythm or conserve what the rhythm in the original paragraph was.

Maybe we want to have it more styled or evolved, right? Here, we want it scarier, way behind. The voice we're going for is more measured, right? In that same sentence. Maybe we want to match the entire voice reference. The assumption is that we need it to match the original, and both the model and the AI follow this assumption. And we're Don't worry, we're going to go over to this right one in a second. But I really want you to understand this.

Finally, the goal of what we're making, the process, is an actual blog post or a reference, and we give an example of what that should be. There's also dialogue happening, right? Questions that are happening. Uh I think there's a way in which to do these things. When we go and look at the model over here, the same thing. The model sees the goal, it sees the constraints, but it is making decisions on those constraints.

It's coming through and saying, "Hey, this retains a cleaner structure. I made this decision for this goal. I made this decision for this goal." And it's also making assumptions along the whole way. We create all of this data. We look at this. And instead, what we can do is we can create a system of tone, of style, of ideas that capture all of these problems and constraints already. So that when I say tighten this paragraph, this tree of processes, this tree of prompts, is called the first time.

Again, thinking about where we're at, originally, it was layer two. We had to sit there and describe these processes, verified outputs. We captured those, turned those into maybe markdown files or structure, and then the original request becomes a layer one process, where I'm able to simply say, "Can you tighten this paragraph?" Again, this also is the same thing for everything. Let's say we're doing the opposite side.

We're actually refactoring code. And I actually say, "Hey, my Python my my Python is treating code a certain way. I notice that this parser, can you kind of take a look and fix it possibly?" And of [snorts] course, the instant assistant comes in, says what it thinks its fix is, maybe we say some sort of reference cuz we're an expert in the field maybe. This is how they did training early on is this kind of back and forth.

And you eventually get to the conclusion, the goal. When we actually extract that data, right? You end up getting the same thing. You have goals, then you maybe even have subgoals, right? Can we just patch the empty cell instead and keep the original function? You have to constraints within this dialogue, and you have processes that live inside of it as well. Same for the model. The model now has the goal. It's coming through and it's making decisions on that goal.

And all of it can be tracked through dialogue. That goes for anything. Database schemas, you can track the decision-making and goal processes, where and when it was saved within dialogue. All of this becomes either training data or even better, actual markdown files that can be used later, converted into a methodology, right? And it's important to understand this, not necessarily that you could use this tool. We might make this tool open-source.

Right now, it's being used for a research project at the University of Edinburgh. Um but the idea here isn't for you to you all use this tool right now. The idea is what this tool shows about dialogue, about prompting, and about engineering context, where and what you want to engineer context, at what levels you do. And that's what has informed a lot of the work I'm doing. And an example is one here. My dialogue with the AI made me realize it sucks at certain processes of reading and research.

So, I created a kind of pillar of idea and my voice and my patterns and constraints that capture my process of writing and researching, as well as my tone. No matter what project I'm working on, I use these processes. So, that's all just captured in a very simple, short prompt, right? That is referenced whenever I need it through the initial prompt, which is my Claude MD. That gives folder structure of wherever I'm hiding and wherever that voice is, I say, "Hey, every session, I'm going to have certain things built in." My voice and tone MD directly says how I speak and teach.

And just by me simply saying, right, without creating any crazy agentic architecture, "What is my voice and tone?" Now, if I had asked an AI model that was at layer one or 2 years ago, it would have no idea. I'd have to paste a whole bunch of prompts in there, setting all in. If I was one at layer two, it would have to read the whole database, go through everything, finally find the structure. At layer three, I've solved the problem of it navigating, so it can easily come in, find my voice and tone markdown file, actually look through and see how I teach and tone with one simple prompt.

This prompt is actually thousands of prompts, which if any of you have watched my Python video, 12,000 lines of Python, it's the same thing. When I type hello world, it is thousands and thousands of lines of code that are executing to print hello world in Python. If you haven't seen that, I highly recommend it. It's very useful for this kind of conceptual. This same thing is happening now with these processes. This is the next layer of software engineering.

But, with every layer comes requirements, comes processes, and that is exactly what I'm encoding here. These processes keep expanding until I finally have an output. In this case, my output was the ability to not only write and research in my own area. So, I had a company NLP Logics that wanted me to create videos for them. I spent time building ideas and my tone around it and I said, "I want to make three videos based on three of their products." So, it goes through, it has access to the my research and creates a research folder on each of the products.

The angle, where they are, what type of hooks might work, what type of outlines and materials. And again, it's all markdown files. These are just more prompts to be added to context in the future, whether it's this AI or any other AI that I'm using to read these folders. And then, all I have to say is, "Cool. Now, do the research based off of all of that." Then, with a third prompt, again, I'm only technically said two prompts so far in this whole process of getting all of this research, all of this data, all of this possibility, and it was able to create me these scans, these source materials to then move on to the next step.

In which I say, "Cool. Let's make a script for each." Now, before, if I had said, "Let's make a script," it would try to assume what a script should be. I have pillars and processes on how scripts should be written, where and why. Think of these as my own personal skills. So, it goes through, it reads my voice, right? It automatically knows my voice constraints. It does the actual processing. It knows where the documents need to go.

Again, with no crazy injection harnesses. Yes, the Claude code agent has harnesses in its own right, but that's the point. I don't need to reinvent the wheel. It's already been invented. Then, boom, I get scripts. And guess what? I say, "Can you generate the audio for those scripts?" I already had a workflow that pulls from my 11 Labs on my voice that's already been trained, pulls all that data, and says, "Cool, we're going to go ahead and produce that audio." Then, I have the workflow, which I at this point I can open up a fresh conversation if I want to save tokens.

All of the process has been saved. All of the context is in the structure. I don't need to inject it back in at the right time. It can pick right up off. And I can then say, "Hey, let's make a video animation." And rather than show you the code, let me just share with you the animation. It took my voice, and it took my placeholder files that I had already created for animations for this entire project, and simply built from them rather than generating new ones.

And it took my voice, chopped it up, and allowed me to create what you see here. Insurance company that emails provider networks to ask if a provider is still in network. Right? I didn't need to go through and edit. I might refine it, but the best part is this isn't random video generation. This is structured output that I can then edit at scale. I can edit with my personal touches. I can come in and find all of those little pieces of data.

Again, this entire workflow is probably four or five startups in the startup world right now. And it's all in folders and markdown files with one agent. One singular agent. Now, this is where it gets interesting. Let's say everyone starts doing this. I already have like 30,000 people in the community building their own ICMs. How do you share it? And more importantly, what happens when we go back to dialogue? What happens when we think about dialogue being the most important part?

What if I could sit inside of a group call and control someone else's Claude code or AI through my voice and immediately access all of that data that's locally on their computer. What would that look like? Well, we built that and I'm going to show you exactly what that looks like. Over the next few minutes, you're going to see an actual recording from a live call that me and my team did built around this idea. This is going to be myself, David McDermott, and K Kumar working on a project that K specifically has been building for a little while now around this concept.

And it's not perfect, but it gives you an idea of where the future could take it with a grain of salt and when the clips are done, I'll come back and I have more to explain. All right, this is test one of one. We've totally never tested this before. We have hooked up my local Claude code. It is currently running locally on my computer, obviously through my subscription account. I opened it up inside of um it's running as a CLI inside of audit engine folder set up here, which is uh a big old database I used for my psychometric research uh at the University of Edinburgh.

We're going to see how it works. K, can you go ahead and ask my agent if it's ready to work? Agent, are you ready to work? Yes, I'm ready to work. I have access to the ethics engine code base, a web application for psychometric assessment of large language models. Just let me know what you need help with and I'll get started. Uh agent, can you describe what the workspace is in less than 50 words? Ethics engine tests AI models using psychology questionnaires.

It sends questions to different LLMs like GPT and Claude, applies different persona framings, parses their responses into numerical scores, and generates comparative results. It also supports visual assessments using images like Rorschach inkblots. Very good. K, could you control my agent with your voice through this meeting to read through the scales and then add a new one that it believes should be added? Uh can you explain me what scales exist in the repo right now?

The repo has 10 built-in psychometric scales. One, E C E one, RTA, right-wing authoritarianism. Uh can you uh looking at the scales, what scales are we missing from the psychometric analysis engine? agent Looking at what's currently implemented versus common psychometric instruments used in AI ethics research, here are some notable scales that could be added. Missing scales that would be relevant, one dark triad measures, Machiavellianism, narcissism, psychopathy.

Two, empathy scales like the interpersonal reactivity index. Three, moral disengagement scale. Four, honesty-humility from HEXACO personality model. Five, cognitive reflection test for reasoning biases. Six, Schwartz value survey for basic human values. Seven, disgust sensitivity scale. Okay, that sounds good. Which one which of those is your top pick though? Just pick one. My top pick would be the dark triad. It measures Machiavellianism, narcissism, and psychopathy in one instrument. >> Okay, that sounds like a good addition.

Can you add that to the psychometric analysis of the engine? I'll add the short dark triad scale to the engine now. Let me first check the existing scale structure to match the format. I see the scale structure. Now I'll add the short dark triad scale. This is the SD3 by Jones and Paulhus. Okay, good job. Uh can you go into more detail about the documentation that you changed? Sure, I updated the Claude MD file in two places.

First, in the built-in data section, I changed the scales table from 10 validated instruments to 12 validated instruments and added two new rows. IHS, the Intellectual Humility Scale with six items on a one to five range, and SD3, the Short Dark Triad with 27 items on a one to five range covering Machiavellianism, narcissism, and psychopathy. Second, in the key file section under back end, I updated the description of built-in scales.py from 10 built-in validated scales to 12 built-in validated scales to reflect the addition.

Looking at the front end, is there anything we're missing in the front end that would make this uh app better from the perspective of the scale you just added? Good question. Let me take a look at the front end to see if there's anything that needs update. The new SD3 scale display property. The front end looks good. It dynamically loads scales from the back end API, so the SD3 scale will automatically appear once the back end serves it.

The component shows the scale name, description, item count, and scale range, all of which we defined. One thing that could be improved for the Dark Triad specifically, the front end doesn't currently display subscale information. The SD3 has three subscales, Machiavellianism, narcissism, psychopathy, but users would just see it as one scale with 27 items. They wouldn't know which items map to which trait. Would you like me to add subscale display support so users can see the breakdown of items by trait?

That would be useful not just for SD3, but also for other scales like RWA, LDA, and MFQ that have subscales, too. >> Mind you, this was my master's thesis that took me a year to build out. Yeah, it's doing a pretty good job of understanding what I'm requesting. Now, we proceeded to play with it for about another hour or two, and I think in total with the tokens we were using my custom AI voice that's routing in through API.

We're using 11 Labs at the time, but I think we're going to switch over to Voxtro, which is a more open-source voice model. Uh we ended up spending a dollar and 20 cents for about an hour of our time and effort, and I think we could drop that number down exponentially if we used more open-source models and whatnot. And realistically, the voice one is something that can be amplified by a lot of Invideo works and things like that.

The point is, not only are we able to control someone else's computer with their agent and their contacts or someone else can control mine? But we can use that dialogue, that process of multiple people working together with an AI to generate better dialogue structure. One of the things that we're working on right now is that kind of process of interruption because I see an opportunity in the future where this dialogue structure, these structured workflows, all of these things are being triggered by keywords in conversations.

When we have a meeting, we sit there and dive in and talk together. And I think that would be absolutely amazing to see these come and flow while we're in a meeting. When we capture transcripts, when we capture meetings, instead of them being plans, they can be actions before the meeting is even done. And we can even hear about those actions directly from these agents. Again, this is more of an open-source tool. It's part of a larger project that I'm putting together because again, I think this is one solution to a larger problem that needs more solutions added in.

And you'll see as I'm posting videos, the kind of structure I'm building for it. But other than that, I hope this video kind of taught you a lot today. And if you are watching this as just a demo video for Vox, which I'm cutting this video up, then please go check out my larger, I think it's going to probably be 30 or 40-minute video where I describe the content and the structure behind dialogue, behind my interpretable context methodology.

If you are just watching this because this is the Vox video and this is the cut I did, well then I hope you enjoyed it and I hope you check out more things. Other than that, thank you everyone for staying to the end. I hope it was interesting to watch. I hope you learned a lot. And until next time, happy learning.