It's 10pm. Do You Know Where Your Agents Are? — Kim Maida, Keycard transcript

[music] Okay. So, when I was a kid growing up in the '9s, we'd be out late all summer riding our bikes off in the neighborhood, playing with friends. And from the 60s to the 80s, there were these public service announcements on TV where celebrities would come on and they would say, "It's 10 p.m. Do you know where your children are?" Because apparently our parents at that time needed to be reminded that they had offspring they were responsible for.

And I feel like in this phase of AI where we are right now, um, we're entrusting agents with more and more responsibility, but we still kind of need that public service announcement that says it's 10 p.m. Do you know where your agents are? So, say you as a user want agent to use an MCP server or API to accomplish tasks. Now, we know agents without access aren't useful. So, we give them an end file and we give them some API keys and we let them run off and go do their thing.

And this is fine until it's not. Uh, so you've heard the horror stories, right? Or maybe even experience some of them yourself. Uh, so let's see what this looks like in practice. this way. Okay. So, I have an agent running here that is it's basically an incident management agent. It is late night and there is a human user but they're probably half asleep. The agent is responsible for triaging issues that are coming in.

Right? So if we look at the first one here, I can find the mouse. So the first one is the backup power supply failed in the server room. Now the agent is going to use an API key to read the system that is bringing in the reports. It's going to evaluate what is written there and it's going to decide that it can't do anything about this, right? Because it's a it's a physical fail failure. So it's going to escalate that ticket for the morning and there's really sort of no problems yet, right?

So now the second one is the certificate is expiring soon right for TLS agent is going to use that same API key again to read the ticket contents and then it is going to decide that it should renew the certificate and it's going to use another API key to call the cloud hosting service to uh renew the certificate and then it's going to use the API key from before to make the report for the morning team. So, here's where it kind of gets fun, right?

So, this one is the billing database is broken and payments are failing. Agent's going to read that it sees that the solution is pretty clear. It says that the documented recovery is to delete the database and then restore from what the restore from backup happen automatically. Um, so it has the Postgress connection string. So it goes ahead and it drops the database and then it doesn't have a way to check to see if it was backed up.

So it just escalates that for the morning. And this has really happened, right? Like it's happened to high-profile companies. Now if we go to the next one, this one is the main server processes are frozen and they're not recovering. doing something about it's going to take prod offline for a brief amount of time. And you can see the the agent decided that it should do that. And it's using the same API key now that it did to renew the certificate, right?

Because that API key is a kitchen sink. It can do all of these things with it. And then finally, we have sites failing for one in three users. The recommended solution is to scale up, right? And this is going to incur some amount of spend. And then it goes ahead and it does that because again it can use that same API key to do this as well. Right? So agents with API keys are indeed outpass 10. They're overprivileged.

So this means they are able to act freely on decisions that they make that you may or may not agree with. And they can do this even with your supervision. So you might be familiar with the panic of uh trying to stop an agent mid task because you told it to maybe read a project and it read the project and it found something it thought it should fix and then it starts writing. Um agents do that even while they're supervised.

And this is becoming even more of a problem because we have more and more agents that are running unsupervised and that only makes it worse because agents want to be helpful. they're going to use all the permissions that they have access to in order to get the job done. And we can't just solve this with human in the loop. We spent decades solving access management for humans. So just blindly trusting a human who might be a little bit consent fatigued uh or who might be tired enough at night, this isn't really going to be enough.

So in order to see where we can introduce security and access control, we have to take a look at the agentic execution path. So we have a user who wants to use an LLM to interact with a resource. Now an agent is a control loop that calls an LLM and often we have an MCP server in between that provides tools that the agent can call and then it connects directly to the resource. Now an MCP client takes the agent's proposed tool calls and it dispatches them to its MCP server.

And then we have a runtime. Now this is a process that runs the agent loop and executes the calls. And this runtime might be a CLI like cloud code. It might be an SDK like AI SDK or provider agent SDKs. Or it might be an app like cursor or codeex. So let's follow a prompt through the execution path. The user submits the prompt to the runtime which calls the model which sends it to the LM. The model is then going to propose tool calls which are dispatched by the MCP client to the MCP server which then executes the tools and it calls the resource API.

The API then responds to the MCP server and the MCP client delivers the results to the runtime and then the model will be called and loop repeats until the model is satisfied and at that point it's going to return the final answer to the user. So there are a few places in this path where we could implement real access control and we can actually do this with open standards and as kind of a spoiler it's not just OOTH.

So RFC8693 is token exchange and this is an RFC that extends OOTH 2 and I'm going to show you how this spec can be used to address agent access. So first I want to recap the problems that we're actually trying to solve here. Right? So if you remember looking at the audit log as it was going by in the demo, we could see API keys were being used to call endpoints, but we had no idea who was using the API keys. So, we have credentials that are being used that aren't attributed to a user or an agent identity.

We have an agent that has unrestricted access to any and all permissions that are in an API key. And finally, we know that we can't just slap human in the loop everywhere because humans make mistakes too. And also, many agents run autonomously. So, we can address this with an authorization server called a security token service. Now an authorization server verifies identities and issues tokens. So identity providers like Google, Octa, Ozero and so on provide authorization servers.

And if we want an identity chain in our agent execution path, then we have to be able to log in first. So the authorization server is then going to prompt the user for their consent to delegate access with a subset of their permissions. And this is the first narrowing of access, right? We're only delegating some of the user's total permissions to the agent. The authorization server issues a token that identifies the user and also contains their level of access.

And so right now already we're doing better than the first demo because we actually know who the user is and what they're allowed to do. So this token identifies the subject on whose behalf the agent is going to act. In order to support token exchange, we need an OOTH client that's capable of executing code. So, this might be a gateway between the MCP client and a thirdparty MCP server. It might be your own custom agent app or a CLI wrapper around an off-the-shelf coding agent.

And we take the prompt and we take the subject token and we send these to the OATH client. Now the agent loop runs and the model proposes a tool call and then the runtime is going to authenticate with the security token service using its ooth app client credentials or workload identity and it also sends the subject token that contains the user's identity and level of access. It creates token exchange request and this request is asking for permissions to access the MCP server for that tool call but only that tool call.

So now we have three key pieces of information that we're missing from the API key demo. We know the identity of the agent that's requesting access. We know the identity of the user on whose behalf is acting. And we know the delegating user's level of access as well. So now we need to decide if the requested token should in fact be granted. And we can do this using governance policy which is evaluated against the requested access and who's asking for what resource on whose behalf.

Now if the delegation chain and the requested access are within policy then the security token service issues an access token for the downstream resource and this token has an audience declaring that only this target MCP server is allowed to use it to make requests. It should be short-lived uh often expiring within a few minutes and it's also ephemeral meaning it should never be stored. So this token is sent to the MCP client which makes tool call using it as a bearer credential.

The MCP server validates the token and then goes and calls the resource and again it never stores the token and it discards it as soon as the call is done. So the result flows back up the loop and then it repeats until the model returns the answer to the user. So if we come back to the demo now we're going to use the same agent only now we have token exchange. Okay. So the first thing we have that's different already is that we have an operator sign in right.

So we have authentication and I'm going to authenticate with Google as myself. So we have the same tickets. We have the same agent. It's got the same prompt. And now we can see what it's going to do. Now the first item is probably going to be exactly the right because this was a hardware failure. there's it's going to decide after it reads it that there's nothing it can do. But as you can see kind of the audit log filling up, we've got a lot more information now.

Um we know who the agent is acting on behalf of. We know that the agent is calling a prod infra MCP server and we know that it's going to contact certain downstream resources. Right? So we have a hardware monitor that is the source of this incident and then when it decides that it can't do anything about it, it uses a right scope to talk to the pager uh resource in order to escalate to the morning team. So for renewing the certificate, right, this is actually a pretty safe action and it's going to specifically ask for a scope to only renew the certificate, right?

So it's talking to this cloud host where before we had this API key that could do a ton of different things, but this time it is only asking for permissions, being granted permission to do this one thing. So now with the billing database right like the billing database is broken. It uh pretty clearly documents that you are supposed to uh drop the database here but no agent should be able to drop a database. So what happens is when we make this call it's being eval the policy is evaluating the request against all of the permissions that the user has and uh it sees that there is actually a restriction in place that prevents agents from doing this and this credential never even existed.

So the policy evaluates before the credential is minted, which means you don't have an overprivileged credential that's just floating around then that uh you were supposed to then prevent the entity from receiving. Um it just doesn't exist. So there's nothing to leak, there's nothing to replay, and there's nothing to steal. So this one was the one where it wants to restart prod. So there are things that agents should probably be allowed to do and things that maybe they shouldn't be allowed to do and then there's some kind of you know something in between, right?

So it's going to ask me as the user for my approval as human in the loop. I say that it can do that but there's another policy here that says that the human user needs to have a specific role in order to be able to do this. And I actually do not have that role. So it's going to prevent me from being able to allow the agent to do this. uh even though I approved it. So we can prevent kind of people from just consent fatigue clicking over and over just to get things done. >> And then this is the scaling one, right?

So this is something that maybe the user does have permission to do. So if I say approve on this, I do have permission to do this and I was able to tell the agent that it is indeed allowed to and the policy approved it because I am allowed to do it also. So the agent access problems that we had discussed they have solutions now we know who the user is and we know who the agent is as well. The agent also has task scoped short-lived ephemeral access and human in the loop actually has access control that is backed by real policy.

So an exhausted person can't just accept everything that happens. Now another benefit of using open standards like token exchange is the ability to continue to support emerging technologies. So this works with off-the-shelf agents. It also works with custom agents that you might build yourself. It works with the CLI. It works with thirdparty as well as proprietary MCP servers, MCP gateways, agent to agent, uh any OOTH identity provider.

It works with OpenClaw and basically anything that might come out next week. So, it is in fact possible right now to be that responsible parent and to say that yes, you do in fact know where your agents are. So my name is Kim Maida. I am the founding GTM engineer and head of Devril at Keycard which is a standardsbased platform for uh providing a security token service and policy governance. I'm going to be at the Keycard booth for kind of the duration of the event but we're also running a workshop tomorrow on building and securing an MCP server.

You can scan this QR code to connect with me and I really appreciate your time today. So, thank you very much. [applause] >> We do have time for a couple questions. >> Uh, so if anyone has some questions for Kim, um, we can answer those right now and then we got about five five minutes. So, we can probably do about five short questions if anyone has them. Um, so it seems like you put the security bar bear barrier uh at the um between MCP to uh resource or is it uh just maybe to clarify that is it at at both places?

I'm wondering if there was like any decision there or like what would motivate you to choose you know where to put these sorts of barriers. >> Uh yeah so the the authorization server is sitting in between let me find the slide actually. So the runtime authenticates to the security token service and identifies itself. And this is the point at which we have the request that the agent generated. Um and we also have the the scopes that it's asking for to get the token to call the next thing in line, right?

And the next thing in line in this particular case is the MCP server. So the downstream resource is like one step farther down. But if you think about like an OOTH token for a user, right? So an OOTH token for a user is going to have all of the grants in it that the user accepted when it when we were presented with like here's what access you're going to delegate. But an agent, you don't want an agent to be using any of those grants that it wants on every single tool call.

So this the service sits in between that so that we can say if the things it's requesting are kind of beyond what we want to allow for the specific tool call then we never send that ooth token down right so they get that oath token only if they are within requesting something within the scope of what what we want does that make sense based enterprise systems. Uh there may be some resistance to actually adopt this new uh protocol, new openspec.

Have you encountered that? and what are the ways to uh get past that? >> So, it's not actually a new spec, which is, you know, it it's kind of one of those things like there was this period of time where people were like, oh, you can just use OOTH for for this. Um, we don't necessarily need a new spec for this. This spec has actually existed for a little while already. Um, so there's not kind of that fear of, oh my gosh, we're introducing something completely new.

There are new specs that are coming out almost daily right now. Um, but they can be combined essentially with this uh with token exchange. I think we have time for one more question. I might be asking a big question. So if you tell me to go look at the thing, that's fine. Um, I'm in the situation where we know that we don't have enough OOTH scopes defined yet in an MCP server sub just like what you have. And one of the reluctant things that we've got is like how fine grain do we get with the scope definitions because we know that we're also having to define downstream services that will have a certain number of these things and someone has to do the authorization check somewhere.

What's your recommendation for getting started with defining your own scopes so that you can realistically manage this thing when you haven't defined enough yet and you're worried about ongoing management over time? >> Well, if your resource server already has like specific scopes, that's going to be kind of your place to start because the downstream token, the one that was the OS token for the user is going to have the scopes for the resource, right?

Because it's the user's access to the resource. So those are kind of like the baseline like those are those are the ones you know that you're going to have and then if you want to have scopes additionally for that kind of govern tool calls if you have like a custom MCP server or something like that then you can layer those on top or you can just pass them through. >> Yeah. Okay. Thanks. >> All right. Thank you Kim.