You Built Your AI Second Brain. Now What? (Here's How to Evolve It) transcript

I've spent this entire year building my personal AI second brain and teaching you how to do the same. I have so much content on my channel covering how to build this system where you have an agent that builds a knowledge base over time with everything that is helping you with in your business. Now, this graph view isn't actually practical. It's just a good visualization to give you a glimpse into how much I'm using this every single day.

I've gotten it to the point now where I'm using my second brain to help me operate in every part of the business I wanted to. Now, don't get me wrong. My second brain is far from perfect, crazy how much time it saves me every single week. Now, the last time I actually measured it, it saved me about 20 hours per week, and that was a few months ago. And so, I really love the state that my second brain is in right now. But, the next question that I have is how do I take the second brain and turn it into a team brain?

Cuz I want the productivity boost for me to extend to everyone I'm collaborating with. It's the big question for me, and I'm getting this question from a lot of others as well. It's the next natural step for your AI second brain. Now, you might not realize this thinking about it initially, but there is so much that has to go into creating a team brain. And so, that's why I wanted to make this video to show you how I'm making this evolution and what my strategies are, at least at a high level.

Because, I mean, even the first thing we have to think about here is scale. You have information coming from so many different places when you have a lot of collaborators. So, how do we get all that information into our brain, and how do we store it? Because also, an Obsidian vault with Markdown is not going to cut it anymore like it is for just you. And so, also thinking how do we expand the retrieval strategies to make it accurate at scale, how do we handle source citation, and then another big thing that is hopefully obvious to you is that we need to handle security and permissioning.

It is very unlikely that your team brain is going to have everybody with full access to all information. You need some kind of separation where we're going to label things at ingestion depending on the source, and then setting up keys for access so we can enforce in the database only certain people can view certain documents. And so at a high level in this video, I want to show you how I'm doing all of this. And for everything that I demonstrate in this video, I have a working example here that I'll link to in the description.

You can even give this to your coding agent to help you incorporate these ideas as you evolve your personal system to something that is team-wide. And let me tell you, this is worth investing in no matter how big your team is. A couple of people all the way to an enterprise level. And I actually partnered up with Oracle. I'm using the Oracle AI database in this example just to show you how this is possible to scale to an enterprise level.

They give us everything we need to help us with our ingestion from our sources, our retrieval strategies, and the role level permissioning. But as a whole, what I'm covering in this video is very high-level. It's going to apply no matter the tools that you use. I just have some recommendations that I'm showing in my demo here. And so with that, let's get into the most important architecture decision first. You might think that in building a team brain, you're going to entirely replace your personal system, but that isn't actually the case.

And I hope that's good news for you because it means that everything you built up until this point is not going to be scrapped. In fact, it's going to be the layer that you continue to interact with. The team brain is really more just the knowledge base that we access. And I actually wrestled with this for a while because there are two paths you can go here. Either you maintain the agent at the personal level and it's just the knowledge base that you distribute.

Or you can make it so that even the agent itself is just a single brain that everyone accesses from their own interface. And there are many reasons why I've decided to have this separation here. You can think about it like we distribute the policy and the knowledge, but we still maintain the personal agent with the personality and the part of the memory system for that individual. This also scales the best because it gives us distributed agents and it allows us to continue to customize our personal system, which that's important to me.

It probably is to you as well. As you built your own AI second brain, I'm sure you realize that it's quite a bit different from everyone else's. So I want you to be able to maintain that. But we still need everything centralized for the knowledge base so that we can manage the permissions, right? We can't just dump all the sources like Slack conversations, GitHub repos, documentation into some GitHub repo that everyone clones locally.

We can't have some Obsidian vault just sitting next to each brain like we do for our personal system anymore. We need a central database, which is what I'm using Oracle for, to store everything and have the single source of truth for our retrieval strategies and our permissioning. And then each individual brain is going to access this single table through an MCP server. I'll show you what that looks like in a little bit as well.

Now, as promised, I'll stay high-level, but as we're going through the different strategies here, I do want to give you glimpses into how my working system functions and how I'm using the Oracle AI database here. So the main thing is I have a single table called documents. Keep it simple, stupid. There is no reason to have a ton of different tables for all of my sources. No matter where the information is coming from across my business for the company, I'm boiling it down into this schema.

So a Slack thread, GitHub repo, documentation, whatever, it comes into this format where I extract the title, text, URL, author. You can see some of the columns here, which will also give you an idea how we can do our permissioning and figuring out, you know, like who's the expert on this thing in the company. There's a lot that I have set up here, but the important thing is it's a standard to keep the querying simple.

We want a very basic interface from our personal agents to connect into our database. So now getting into the implementation very quickly here, everything that comes into the knowledge base comes in under this single format of a document. And so the schema that you're looking at here maps directly to what we have in the Oracle AI database. And so now each of my connectors, like the Slack connector for example, its sole responsibility is to fetch the information from the Slack thread or whatever, and then mold it into the format that fits the document, right?

So figuring out like here's how we fetch the author, the domains, the metadata, the source from a Slack thread, get it into this format now where we can put it in the knowledge base, and now we have a single entry point for our individual AI second brains to query. And we're just going to be doing that through a pretty simple MCP server. And the MCP server, even though it's simple, is very important because it is the connection point between our second brains and the company-wide knowledge base.

And so it's just another tool that you give to your AI second brain. You don't really have to change anything in your actual architecture. So we have a tool to identify who we are, just to confirm our identity, one to search over the documents that we have access to, same with code cuz it works a little bit differently, and then one to fetch an entire document again, assuming we have access to it based on our permissions and the domain for that document.

And then another one that I've been experimenting with recently is a who knows function. This is cool cuz if we attach an author to individual records in the knowledge base, we can also identify like if we have a follow-up question, who's the person in the company that we go to. So yeah, I hope that your your gears are turning your brain right now. All the useful ways that you can build functionality into this MCP. Okay, so you've seen the database, you've seen how we organize things in the MCP server.

Now let's get more into the pipeline and talk about how this works and why we need this in the first place. So obviously the big gap being if we don't have any permissioning, then a query through the MCP server is just going to return all records, and we definitely don't want that. But the first step, even before we can set up the permissions in the MCP server, is we need to label things at ingestion. So when we're bringing sources into that single documents table, we need to know that like this came from the ops slack or this came from the marketing wiki space, so we know who should have access to it in the first place.

And I don't have to get technical here at all. It's pretty simple. When we set up our sources initially, we're just going to create that mapping like this specific source is going to have this label. So, we just apply that to every single record every single time we're entering that new document into the knowledge base like we're looking at right here. Like this came from the marketing slack, so it's going to have the marketing domain or it's going to have the ops domain.

And this right here, this column is how we're going to apply the row level permissions in the Oracle AI database to filter out what certain people can access. And of course, if you have spaces where multiple groups are collaborating like ops and marketing, of course you can apply multiple labels to a single document, so you can expand who can access it. And that's really it. We just have that column for the domains. That is one side of the permissioning.

The next side is we have to create the key and have the MCP server use that so the database can identify who is trying to access and then filter records based on that. So, within my Oracle AI database here, I have an MCP tokens table. Super simple. You set this up ahead of time, right? Like some kind of administrator in the organization is going to create these tokens and then assign them to specific people in the company.

And so then you give that token to the individual, they set that up with the MCP. I'll show you what that looks like in a bit. And now that is our ticket into the knowledge base. And so with that, I'll also show you I have a principal table here. And so this is how we map the token to what they have access to based on it. So, principals, let me run this query. So, right here we can see that like for each user, we have a mapping to the groups they have access to.

Like Jeff can only access the parts of the knowledge base for the ops domain. Julia can access marketing and ops. And then obviously most of the time you're going to have leadership where you'd want them to to able to access everything. So, a special tag that the permissions will recognize here where leadership has access to all knowledge that we have in the documents table. So, once we have the key set up, then when we make a query with the MCP server and one of those keys for our authentication, now we're going to enforce in the database.

So, we ask our question like what's our discount ceiling? And I'll show a live demo of this in a second. Then, the database is responsible for resolving the identity using the key to figure out the domains we have access to and then applying that at a row level. So, it's a row level security we have built into the Oracle AI database. The most important takeaway here, no matter how you build this system, is you need the gate to sit in the database.

You cannot have this second brain, the personal part of the system, responsible for the security in any way cuz then it's going to be way too easy for the individual to get around it with, you know, sort of like prompt injection to their own agent or just changing their own agent's implementation. We need to have the access be determined by the system that is running remotely. So, for the sake of our demo here, I am seeding the users and their access and MCP tokens.

Generally, yourself or some kind of admin in the company is going to get these tokens created initially and then distribute them to all the individuals. And so, then they just have to have very simple MCP configuration. This is all it is to attach their second brain to the knowledge base through a remote MCP. So, I have this running locally for the demo right now, but usually this would be a remote address and then you set the authorization to the token that you were given.

This determines all the access. It is so simple to get your second brain connected into this knowledge base. That's the beauty of how we've set things up here. So, no matter how you use your second brain right now, it doesn't have to change. We're just attaching the MCP as an additional tool. So, typically I use my second brain within either the Codex or Claude Code CLI. And so, here I just ask a simple question. What is our enterprise discount ceiling?

And based on the question, my second brain knows that this is something that we need to access the team knowledge base for. So, it's going to use the team brain MCP. It's going to run that query. And if [snorts] I do a {slash} MCP, you'll see the five tools that I showed in the code very briefly earlier. So, it identifies the tool, it calls it, it runs the query, and then we get our answer. Reps can offer up to 20% off the list on an annual contract.

And specifically, Jeff, who I'm using the MCP as, he has access that information in the knowledge base. Or sorry, I actually ran that specific demo as Sam. Because you can see if I ask that question with Jeff, it says the knowledge base doesn't cover anything around the enterprise discount ceiling. So, the information is completely invisible as it should be cuz we can't access it. But if I ask the same question as Sam, this is where we get that up to 20% off list price like we saw within my second brain Claude code session.

So, completely invisible or you could have some kind of system where like it knows the document's there, but it says it can't view them. If you want Jeff to know like, "Okay, I would have to go ask Sam." So, couple different ways you can set it up, but it'd be easy to tweak this if you need. So, by default, no match means no access. And that's when you know you have to ask someone else about it. So, you can have a separate part of the system like I set up where we have the documents, yes, but we also have a separate place where we identify who is the expert on certain things.

So, I can literally just ask the MCP like, "Who knows about the batch cluster auto scaler?" And Jeff is the one who knows this with Elena as the secondary voice in the Slack thread. So, it dug deep into the context there. And so, if I wasn't Jeff, if I was Brian asking about ops stuff, then I would know to go and ask him. Cool. So, the MCP server is the star of the show because it's what makes the connection super easy here, handling all the permissions.

But the other really important thing we have to talk about is the retrieval strategy. And I want to show you how I'm doing this with the Oracle AI database and LangChain as well. Cuz here's the thing. We're not operating in a personal database of markdown anymore. We have a dedicated table. We're potentially consuming thousands, millions of records from Slack threads and repos and things like that because we're evolving to more than just one person.

We can't just do a Karpathy LLM wiki with just keyword search anymore. The best strategy that I found is to combine keyword search and semantic search together. And they kind of cover each other's flaws, right? Like keyword search is able to find very specific wording or IDs, things like that. Then semantic search is able to find meanings, concepts that are related that don't actually have the same keywords. And so, both together in a single search and waiting the different results that come back, this is what's optimal.

And this is really important because even with the permissioning, there's still going to be thousands of records potentially we have to deal with. Like let's say we have 40,000 in our database minus what the database won't show us, so maybe that brings it down to 10,000. Well, we still only want a couple of documents or snippets that we give into the agent to deal with and give us the final answer. So, it's definitely important to have a powerful search strategy to cut out all the fluff and give us just what we need.

And of course, citing sources as well, so we're able to even go back to the source in a Slack thread or documentation or whatever to verify things. To just know that the agent isn't making things up. This really came from something in the company. Now, implementing these search strategies with the Oracle AI database is so straightforward because they have an integration with LangChain so that we can very easily use the LangChain Oracle DB package to create our embeddings.

And we have the retriever, so we're able to do the keyword search and the semantic search. And Oracle AI database supports all this out of the box. Like really powerful keyword search across our data. They have embedding models that we can use directly within the database. Everything is out of the box for us to build exactly what we need for the team brain. So, with the LangChain integration and what Oracle AI database gives us out of the box, we don't have to bring in anything external.

Embedding models, libraries, and nothing. We have what it takes to embed our documents and user queries, retrieve things with vector search and with keyword search. There's also really good documentation for this integration directly in the LangChain docs. If you want to incorporate this for yourself, give the documentation over to your agent. You can even get started for free with the Oracle AI database to experiment with these ideas, run my full working demo as well.

But overall, I hope that you just got a lot out of the high-level strategies here. No matter how you build a team brain, this is what you should be shooting for for evolving your search strategies, building in permissioning, the super simple connection with the MCP server. Use this as a template to get started with your own team brain. And so, if you appreciated this video, you're looking forward to more things on AI second brains and agentic coding, I would really appreciate a like and a subscribe.

And with that, I will see you in the next video.