The Biggest AI Coding Agent Upgrade Is Already on Your Machine?! transcript

No matter the coding agent that you're using, like Claude code, and what you're using it for, there is a goldmine of data that you're probably not taking advantage of at all. And in this video, I'll show you how we can use it to make our coding agents better and better over time, and it really doesn't take that much effort on your part. The data that I'm talking about is the conversations that you have with your coding agent, plain and simple.

Every single conversation gets stored as a file on your computer, no matter the coding agent that you're using. With Claude code, they are stored as these JSONL files. And whether you know it or not, your coding agent has probably already looked through these files before. Even if you have some kind of memory system set up, I find that when you ask about past conversations, it likes to look in these files because they are rich with information.

That's the whole point of this video is we want to leverage these, and I'll talk about how we do that. And so, the problem we have with this though is that these files are so rich with information that it's kind of overwhelming. So, even if you've looked at this before, you've probably scrolled through it and been like, "How am I even going to use this for anything useful?" Well, that is what I have for you right now.

Because what we can do with these beautiful files is we can take a look at the hundreds or thousands of past conversations with our coding agent. Each one of these files for that conversation has every prompt that we sent in, every tool or MCP server coding agent has used, every thought that it's had. And so, we can take all of that, extract the good parts, put it in some kind of structured table, and then start extracting insights.

Like we can identify opportunities to make our coding agents more token efficient, or figure out common failures so we can address them with changes to our rules or our skills. Your imagination can really start to run wild with all the different ways we can leverage our past conversations, and of course, using our coding agent to help us do that. And so, what I have for you here is such low-hanging fruit because the input into the process I'll show you is just the conversations that you already have with your coding agents stored on your machine.

There's nothing else. And what you get out of it is the insights to make your coding agent better. So, we are using our AI coding assistant to learn from past mistakes to make itself more efficient and reliable. The main thing that we need to make this process work well is a place to store our transcripts permanently because otherwise Claude Code cleans them up after 30 days by default. And then we also need a database for structure.

Like you saw just a bit ago, the JSONL files are super messy and it's not better for any other coding agent. And so having a database where we can split things into structured tables, just getting the information we need. Like here are our sessions, here are all the tool calls the agent has made across conversations, here are the different agent turns. I'll show you how to set this up here in a platform that I love using called Data Bricks.

This is my favorite free solution for everything I'm going to show you in this video. But there are of course a lot of different ways we can extract data from these conversations and get these insights. This is just the way this worked the best for me. Okay, so let's start with the easiest way to get this done. Now this isn't the best way to do it, which is why I want to show you the process with the structured tables.

But if you want to get started simple as possible, this is all you have to do. You just go into your coding agent, couple of sentences here. Do a deep dive into our past conversations to identify opportunities to make our coding agent like Claude Code more efficient or reliable. And then I want you to suggest the top 10 improvements in a concise bullet point list. So obviously you don't have to prompt exactly as I do, but I like adding this just to make sure it doesn't spit out change after change that it recommends.

Cuz especially before Opus 5.5, like Opus 5 would just give me like a million different things that barely mattered at all. And so I like to very much bound what it outputs. And this is all it takes to get started. It's super easy. And I also started this conversation by showing you that coding agents are aware of where their own conversations are stored. So if you just simply ask, "Where are all Claude Code conversations stored on my machine?" It is able to identify that exact folder that I was showing you all the JSONL files in earlier, which they're all split by the different projects that we're working on with our coding agent.

Or if you just search the folder for star.jsonl, it'll show all the files. We can copy them all if we want to. Now, back over to Claude code, once I send in this prompt, it's going to start by loading a skill that I built for this exact process. So again, this is the simple way to do it. It's not the most effective or token efficient, but if you do want the easiest starting point, I will link to this skill in the description, and you can just send off a prompt like this.

And so it's going to run some shell commands to read through all these files here and identify common failures. And so it's going to list out the top 10, and then obviously I can take that forward and then ask Claude code, "Okay, like based on this failure or this failure, what can we change in our AI layer, like our hooks or our skills or our rules, so we're not going to run into this time and time again?" So for example, with the third issue that it identified, it said that I'm hitting my concurrent sub-agent limit way too often.

That is definitely a problem that I want to address. And so taking this forward is as simple as asking it, "What can I change in our AI layer to fix issue number three?" And by the way, my definition for AI layer is built into the skill, so it knows that it's doing a self-audit of its sub-agents and hooks and rules and skills, everything that you've built into, you know, your global rules or your .claude folder. And so here it makes a suggestion that we should make a change to this sub-agent.

This is the culprit. Assuming that it's correct here, we make this change and now future conversations won't run into issue number three again. And so it's all up to you just continuing to work with your coding agent in the future and seeing if you really did address this problem. So at a very simple level, this is how we leverage past conversations to make future ones better. But now we get to the real system here, because the problem with doing this the simple way, as good of a starting point as it is, is we are asking the coding agent to go through and read hundreds of thousands of tokens of files going through these massive JSONL files that I was showing you earlier.

So, if we go through this list, either it has to read everything to identify all the right patterns, or it's only going to pull the surface level things and just pick out a few of these files to read. Neither of those are ideal. And of course, there are other open-source projects out there like CC usage and Claude Mem. It's specialized memory for your coding agents, but these two and really nothing else that I've seen is really directed towards storing memories specifically so that we can make our coding agent more efficient and reliable over time.

That's what we want to do right now using Databricks. And so, I'll show you how easy it is to install it, use it for free to build up our structured data set for our conversations, and then start extracting the insights from them. And I'll show you how within the Databricks platform, we can use an agent they have called Genie to help us define the structure and build up these tables. So, you have a conversation like this, so it helps you create the structure and gather the insights, so we don't have to do anything manually.

Again, this is low-hanging fruit. Either this way or the super simple way, we really don't have to do that much ourselves, especially considering how powerful the insights are that we get, saving us a ton of time in our conversations going forward. And full disclosure, I did decide to work with Databricks to bring this video to you. I thought it made sense because I'm already using their platform to do this, and their free edition gives you everything you need to go through this entire process.

All right, so I'll start by quickly showing you how we can install Databricks in just a couple of commands and get it connected to your coding agent of choice, so it's very easy to work with. And so, we're going to be installing the Databricks CLI. So, everything I've been showing you in their user interface, we can set up on our system. So, I'll link to this page in the description that has installation instructions for every OS.

Once you have this installed, then you can also connect it to your coding agent like Claude code. So, I just use the Databricks CLI AI tools install the agent is Claude code. You could do Pi or Codex, something like that. And then just for this project, I'll also have this command in the description. So, you've got it installed, you got to connect it to your coding agent, and now we have it as an MCP server. So, pretty much anything you want to do in the interface, like upload your transcripts or create the structure tables, you can do it directly from your coding agent.

So, for example, within a Claude code conversation, I can say use the Databricks Genie MCP server to query my Claude code history. Obviously, this is once we have things set up, and that's what I'll show you next. But, I'm just showing you how easy it is to work with the platform within our coding agent where we already work. And so, it's the same kind of thing I showed you with the simple demo, where we're having it go through past conversations to identify failures and patterns around that.

But, now we have a lot of structure. It's going to be token efficient, and it's going to be a lot more reliable of a process. And also, just to show you really quickly, we can just use the Databricks CLI directly to ask Genie, which is the agent built into their platform that has access to everything you set up there. And so, I'm running the ask command, and then just giving it a prompt like you would your coding agent.

So, once you have Databricks installed on your machine and connected to your coding agent, we have to go set up the storage of our transcripts and the tables. We can't just go ask the questions right away. So, here in the homepage dashboard of Databricks free edition, I'm going to go to catalog. And so, we'll start by creating a volume. So, for the sake of time, I already have one created, but you'll just go through this process.

It's super super quick, and then this is where we're going to store all the transcripts. So, we'll just upload the conversations we want from our computer right here. And at this point, it's still in the JSON L format. And so, literally all I did at this point is I just had my coding agent upload these files directly to the volume using the MCP server I just showed you how to connect. And of course, you can also upload these manually as well.

Like I can just go to my file system here, select my conversations, drag them in, and then upload them that easily. And so, for this demo here, I uploaded 62 of my biggest conversations, just so I'm not uploading thousands of files here. But you definitely can do that if you want. The main thing I would say to keep in mind here, you just have to be okay with this, is you are taking your coding agent conversations and uploading them into a whole new platform.

But I will say that Databricks, they are not going to train on your data. There's not really any kind of problem, at least in my mind, for uploading your data here, especially because you should not be giving any sensitive information to your coding agent in the first place. I mean, yes, if you have API keys that are stored in these JSON L files, you probably don't want to upload that anywhere. But also, you shouldn't be giving Anthropic or OpenAI or whatever your API keys, either.

So, assuming these files are clean and you don't have anything sensitive here, you can upload it right into the platform. Otherwise, you could have your coding agent do some kind of scrubbing before it uploads with the MCP. So, anyway, I just wanted to show that really quickly, cuz I know some of you care about that a lot. But the point is, once you have your conversations here, you are good to go. You have your volume set up.

And so, now what you want to do is you want to click on the three dots right here and copy the path to your volume, because we're going to give that to Genie, the agent built into Databricks, so that it can look here, pull the data from this, and create the structured tables. So, you copy that path, and then you go to workspace, and you can create a new notebook. And so, we're going to have Genie actually write the code to operate directly in our Databricks environment.

I obviously have a notebook created already. So, I'm going to click into this, and I'll show you what I did to build this with Genie. So, I'll start by clicking on this button on the top right to open up a Genie. This is the agent that has access to my entire Databricks environment. It's like an amped-up coding agent right here in the platform. So, I'm giving it the path to my volume and I'm asking it to process it. So, reading every JSONL file, but with Spark, right?

Like I'm not spending like hundreds of thousands of tokens here. It's going about it in a very structured way. So, I'm saying these are Claude code session transcripts with inconsistent nested schemas across records. That's also part of the problem here. Is every single one of those conversations is formatted in a bit of a different way. And so, that's the challenge that I'm having Genie address here. And it does a really good job.

So, it gets a general idea of the structure. It understands the inconsistencies. And then it works with that to build this full notebook here. So, I didn't write any of this code. I never write any code these days. Genie created everything to process based on the initial dive that it took, getting everything created into the tables that it built at the end right here. So, if I scroll down in the bottom of this conversation, you can see each of the tables that it created.

So, it created turns. It created tool calls and sessions. And so, it says the number of records in each one of them and the columns. And not only does it create these tables, but it populates everything based on all the conversations in the volume as well. So, for my specific process here, I did it in two steps. First, I had it understand the data and prove that out to me. Then I prompted it to create the tables for the structure and load everything into the database.

And so, you don't have to do it that way. It could even be a much more free-form prompt. Like here's the volume, create some structure for me to get takeaways from the Claude transcripts. Whatever you want to do, Genie is going to be able to just run with it cuz it has full access to our environment here. And I don't really want to get into the weeds for, you know, how it reasoned through the data and built these tables.

But it is really impressive. Like, yeah, my prompts are a little specific, but there's not too much guidance that I gave it and it really knocked this out of the park. So, once Genie has created the tables and loaded it with our conversation data, now we can go back over to our coding agent and use the Databricks MCP server to start querying the structured data asking the questions that we need. And so I can say like, you know, what kind of patterns of failures are you seeing here?

What do you propose we change in our AI layer? Exactly like I showed you in the more simple demo. But the benefit here is we have the structure. We don't have to read through hundreds of thousands of tokens of JSONL files just to get takeaways each time we want to do our analysis. And bringing in more conversations is a piece of cake cuz we already have the pipeline. We again don't have to spend hundreds of thousands of tokens to have our coding agent do that for us.

And so with this question here, instead of all those shell calls reading all these files, it's now making the calls out to Genie. So Genie is really the brains that is understanding the data and giving the insights. And now we're using our main coding agent like Claude Code really as the orchestrator to just ask the right questions to get the output from Genie. So now with my specific demo here, there are three core patterns that it found of failures across all the data.

So Genie, it wrote the SQL, it analyzed the table very efficiently, and pulled this stuff out super quickly. And now we can take this and ask Claude Code what can we change in our AI layer to avoid these issues going forward, just like I showed you again with that example earlier. And at the very bottom of the conversation, it even gives me a link here to explore in Databricks. So I can take a look at the conversation with Genie itself, what Claude Code delegated here, so I can get more insights if I want to really dive into it.

But the most important thing I want to show you is the improvements I actually made to my AI layer based on this conversation. I didn't just build this video to show you something without trying it myself. I used this exact process to prove it out to you while I made this video. And of course, I've run the pretty much the exact same process many times before. So anyway, I'll go into VS Code and show you the real changes that I made to my AI layer that I use within my second brain.

And I'm going to keep these going forward. These are legit. And so, the first thing is I added just like, you know, 38 lines to my global rules addressing the specific problems we found with our Genie analysis in Databricks. Like, for example, never guessing a path, which is something that coding agents will actually do a lot if you don't explicitly tell them not to. I added some more permissions within the settings.json, so it's able to work better with Git.

And then, I also added a hook, which is my favorite part of the changes that we made to the AI layer here. It is a session tree. It's a script that runs whenever I start a new conversation with my coding agent. And this is pretty cool. It injects the real repository layout, so the agent is less likely to guess paths to access things when it's reading through files and gathering context. This is awesome, especially cuz this is dynamic.

A lot of people like to give like a code base layout in their global rules, but then you have to constantly change it and it always risks going stale. But now, we're going to gather context around the file base structure and inject that right away in the coding agent whenever we start a new conversation. So, some pretty cool and practical changes that I've done to my AI layer here, all thanks to the insights I gathered from what I set up in Databricks.

And I know these examples are very specific to what I have in my own system, but I just wanted to give you some real ideas, some real examples of how we are making our coding agent better over time looking at our past conversations. So, there you go. That is how you use Databricks and their Genie agent to make your coding agent better over time. And the big thing here is just that your past conversations with your coding agent are a gold mine.

There are a lot of different ways to structure the conversations and gather insights from them and act on them. I just wanted to give you a full process end-to-end here. And so, if you appreciated this video, you're looking forward to more things on AI coding, I would really appreciate a like and a subscribe. And with that, I will see you in the next video.