Hacking Agent Architecture | Bug Bounty Explained transcript

at a very interesting time in security research at the moment where we're kind of at like an intersection between software engineering and cybersecurity. And you know, if you're smart, you're really going to want to take advantage of that. And that's why in this video I'm going to cover basically some of the tactics that I've been using to build my hacking agents. And I'm going to cover kind of like the meta strategies to go over the decisions that you're going to want to make as you're building your hacking agents.

And essentially just some of the architectural choices that I made. So we're going to get straight into it. And also, if you haven't noticed, I'm in a different location that I'm normally in and that's because I recently moved. So I'm just setting a few of these things up right now and the background is kind of boring right now. So it's a bit unfortunate. And I don't have a curtain either, so it's like very bright behind me and I spent ages trying to figure out the lighting.

But yeah, it's very cool to move out and stuff, but it does cause a few issues. So anyway, we're going to get into building your own hacking agent today. So one that's super interesting, obviously very on topic. Now with all like the security stuff going on. In terms of building a hacking agent, there's kind of two main debates that are going on right now. And the main debates are between whether to have like an agentic hacking agent or whether to have one single agent that kind of has a lot of skills and has a lot of tailored abilities.

So I gave a few examples here. And in general, the kind of idea behind this is whether to have one like sort of orchestrator agent that is going to basically spawn in smaller agents and then just kind of coordinate them and like motivate them basically and point them in the right direction. Or whether you want to have a sort of system where one agent kind of does most of the work and it just has a lot of skills and like CLI backing and tools to allow it to do that work well.

And it kind of more relies on skills than agents. And there's a little bit of a debate on this right now, but in general, if you pick like multi-agent architecture, you're generally going to have like better context management because when you have this orchestrator agent, the agent can kind of focus on high-level thinking, while the agents that it spawns, the sub-agents, can focus on doing things like, you know, wrapping through JavaScript and kind of like polluting our context with other kind of non, you know, hacking-related things.

It's basically like, you know, just running tools and CLIs and stuff, while they just deliver the results to the orchestrator agent. And when you have this agentic architecture, you can build more specific skills for the agents, and you can kind of direct them to certain tasks. Like, you could build like a dedicated agent for like JavaScript analysis, say. And when you have this agent, you can give it specific skills about JavaScript analysis.

You can build specific tools for that agent. Whereas, if you have one singular agent, then, you know, you can build one skill, but you also don't want to build like too many or too big of a skill because when it loads that skill, it's going to obviously, you know, go into its context and stuff. So, there is like trade-offs here. But, if you do use agents, then usually it's going to have higher token cost. And this was like this is a document about this from Anthropic, and it's a very interesting research paper, and I'm I'm going to link in the description where they talk about basically using agents versus like one singular LLM.

And they basically talk about it's like three to 10 times higher token cost, which is like quite a lot. And that's mainly because the AI has to like basically redo the same task, and it has to like reread the same files, and that's what causes the increased token cost. And also, when you have like this orchestrator agent, you're kind of relying on the AI to manage other agents, which honestly, I don't think it's very good at.

Like, personally, I found that the AI is not very good at managing other agents from my experience, anyway. And, you know, obviously, if you have the single agent, it's just like flipping a coin at this point. So, if you have the single agent, it's going to have less token cost. And, you know, the agent is able to kind of like, you know, you don't need to manage anything. The agent kind of just works by itself, and it's just going to chase leads on its own.

Makes it a little bit simpler. But, obviously, you know, you're going to have one agent with like one context, and then you're going to have to, you know, build specific skills for that agent. But, you're not going to be able to develop it the same way as an agent where you can actually have like a custom prompt for every one of the agents, as well as like skills, and even like MCP servers being custom installed. So, there is like trade-offs there.

And, I decided to go with the multi-agent architecture, uh mostly because the way I kind of built my tool was hacking specific vulnerability types, and I thought it was, you know, it was better to build the hacking agent that was going to be able to kind of niche down into specific vulnerability types, which is what I decided to go with. And, basically, the other choice when you have your architecture is whether your hacking agent is designed to go wide, so it's designed to go across a number of different programs, or if it's specifically designed to go deep.

So, if it's designed to actually go into one program and discover as many things as possible inside of that program, and kind of dig deep into the features, right? So, originally, my hackbot went wide. And, when I had this going on, it worked quite well. I can't even lie, like the wide hackbot does work quite well, right? But, I eventually pivoted to the deep hackbot. And, I kind of explained why here, and I have a few like points for the deep one.

But, obviously, both of these are viable options when you're building your hackbot. And, honestly, like it's just kind of depends on your hacking style and stuff. But, I decided to go with the deeper option after kind of experimenting with both. And, the main reason I I for the deep option was one because I kind of liked learning with the hacking agent. So, when you will have like this hacking agent that's going deep, it's kind of discovering like gadgets and stuff, and it's going like straight into the features that you've already tested out.

So, I'm running out programs that I'm obviously familiar with. So, when the AI agent is finding like leads and stuff, I can kind of connect the dots in my head and be like, "Oh, wait, that's actually like a really good chain, and maybe it's actually useful for this as well." So, I was kind of like thinking about these things, and I found that very interesting. Where, you know, I deployed a hack bot on targets that I was already kind of in the process of working on, and it was able to like learn with me, basically.

And as well as that, when you have this deep hack bot, you're going to get less, basically, noise when you're kind of looking at the reports. I'm going to talk about more of this later, but when noise is basically like like the random like weird reports that you get from AI that just like don't even have any impact, but they like kind of sound like they do, right? So, that's the thing, like when the AI writes a report, it's like it's actually quite convincing in terms of like reading the report.

It kind of makes you think like, "Oh, wait, maybe this is actually like a real issue." When you go into a domain, you start reading it, it's like, yeah, maybe this actually isn't like that big of a deal. Because it'll frame it like, let's say, it'll be like unauthenticated access to like some service, and like the service name is like, "Oh, wait, that's that's actually a big issue." And you go into it, and it's like it leaks like some demo account or something in the company.

You're like, "Okay, this is actually not that big of an issue then, right?" And that's like a lot of the time where you get or you'll get some like XSS vulnerability, but it's like the person has like to like 20 clicks for it to work, right? It's like, yeah, I mean, it kind of it is an issue, right, but it's also kind of like a non-issue, because it's like, you know, crazy user interaction or something, right? So, sometimes you do get these kind of weird like, you know, these like non-really relevant vulnerabilities that we don't really care about, but the AI still flags them as vulnerabilities, and it makes it very convincing.

But obviously, when you know a lot about a target, and you're basically like familiar with them. When you see one of these reports, you can you can just get rid of them basically instantly. And you'll probably get less of them as well just because there's only like a limited amount that the AI can produce on one target, right? Before it has to actually start doing real reports. And yeah, basically you can just filter out stuff a lot easier as well.

But this is the thing when you go deep, right? It's like it is more work, right? Because you have to kind of teach the hackbot about a target yourself and you'll have to guide it through. You have to point it at specific things, which means you have to be actively working on the target as well. That's kind of the main downside of it is that you have to be actively working on the target that you are pointing your hackbot at.

Because otherwise, you know, the hackbot's going to like it's not going to know what to do, right? It's not going to know what to do with itself. It's not really like thinking about, you know, how to like go into these type of features. It just wants to find like one of the least guys what you told to do, but doesn't know what to do with itself, right? So it's better to kind of guide it when you're looking at going deep, right?

And the other thing is it kind of requires experience. And it requires experience not only like in hacking in general, right? Because obviously you need to be able to identify where to point the hacking agent at, which is a skill in itself. But it also requires experience on the target itself, right? Because you need to know in the target what kind of vulnerabilities are actually here, right? It kind of requires you to actually have vulnerabilities on the target before you actually start hacking.

Because you need to be able to point the hackbot at like high value areas where you've kind of got traction before. And that's honestly the ideal way to have say deep hackbot is like you know that this target was like vulnerable to these vulnerabilities beforehand and you kind of custom tailor this like agent to go after those vulnerabilities. And that's honestly the best way to do it because you just know that the target is vulnerable there or there's like a particularly sensitive feature where there's like a lot that could go wrong.

That's where like I would be pointing my hackbot at because it's just going to do better there. But obviously that requires you to know some things about the target. It requires you to understand how the features work, and it often requires you to have found vulnerabilities in those areas. And, you know, kind of learn from that. So, that's honestly kind of the disadvantage of it. When you're going wide, you know, the hack bot is doing more stuff itself because you can't guide it, right?

So, the hack bot is going off its own ingenuity. And it's basically completely based on the workflow that you've designed. And the AI itself is always going to be the same, right? It's just like a common factor across all of the hacking agents. The difference is how you implemented it. And usually this requires you to build skills into the AI, or, you know, do certain like patterns of vulnerabilities that you found across numerous targets.

And that's basically how you code in into like the AI is you basically building like your own workflow into this agent, and it will go off and repeat that workflow presumably on all of the targets. The main issue I found with this was that when the AI would deliver these reports, well, one, I didn't know what was noise and what wasn't. So, I was looking at this, I was like, "Mhm, does this company actually care about this, you know?" It's hard to know what a company will actually care about, and what services they actually do care about, and, you know, what the actual model is if you don't even know what the company does, right?

Which is what was kind of happening to me, right? And, you know, the AI was delivering these reports, and I was like, you know, "Where Where even is this, you know?" It It mentioned some feature, and I didn't even know where it was. I didn't even know where to find that, first of all. So, it took a long time to kind of start triaging these vulnerabilities and trying to figure out, you know, is this an issue? And then when it comes to certain vulnerabilities that you get delivered, say like gadgets like a client-side path reversal, trying to sort of escalate them is a nightmare because you're looking at this target that you've never seen before, and you're trying to escalate vulnerabilities.

But, it's like, you know, you don't even know where to start escalating. Like, you have to first of all get familiar with the target. You have to figure out that we have to sketch it here. Now we need to turn this into a vulnerability. We have this API that we can hit. You know, it's like when you don't have this sort of backing that you had before, it's very hard to start escalating yourself. Obviously, you can use AI for this to try and escalate it, but you know, it's not always that reliable.

So, I decided to pivot to the deep hack bar, but both are very viable and I'm sure both can make a lot of money. And there's examples of both working very well. So, this is just a choice up to you. And the next part is unfortunately this is a catastrophe, but it's off with hack bots. So, you know, we know what's hackers, right? The best vulnerabilities are always going to happen from an authenticated state, right? That's where you're going to get stuff like SSRF.

You're going to get stuff like RC. Really, it's when you're authenticated is the best. Obviously, unauthenticated vulnerabilities happen. Of course, unauthenticated vulnerabilities happen. There's open source software you can look at as well. You don't need authentication. But, for most bug bounty programs, I'd say like 90% of them, it's like a product that you need to sign up to and the back end is not exposed to you.

So, you don't know what the source code is until you sort of get into it and you start replaying the HTTP requests and you can kind of build a mental model of what's going on behind the scenes. And to do that, you need to have credentials and you need to go and log in to these platforms, right? And you know, if we're building an autonomous hacking agent, well, then the AI has to have the credentials and it has to log in for us.

It's pretty simple, right? It doesn't sound like a big issue. Well, it kind of is a big issue. And the main issue is these very great companies here have decided to make it very hard for these AI chat bots to log into these applications. And these applications are called web application firewalls, WAFs. And they hate us. They hate AI hacking agents because they developed captures and data center blockers to stop us from basically deploying these AI chatbots.

Okay, for real, the systems weren't developed for AIs that had to log in, but they're mainly for crawlers, right? So, these websites will stop crawlers from logging in and basically minting like a load of accounts on these websites, and basically stop them from scraping like their data and stuff. That's the main reason these things exist, but they unfortunately are a huge blocker to our AI chatbots, which probably run on the VPSs, you know, virtual private servers on the cloud somewhere, and these IPs are always blocked, which is so annoying.

And the other thing is if you're using a headless browser, like with Playwright or something, you're 90% chance are you're going to get blocked as well. It's like these systems will pretty much always block you. If you use curl with your requests, you're going to get blocked as well because the user agent is going to get caught. You're not going to have the same like headers that are getting sent in, and it's just very annoying.

So, auth is probably the biggest problem when you're building a hack bot, and this is honestly probably the first thing that you need to think about when you're building it. So, what is the solution to auth? So, the solution is we need to basically stop using this data center IP, and we need to use an actual version of headed Chrome. So, we can't use a headless Chrome. We need to use an actual headed Chrome in order to get past these captcha solvers and these challenges because the AI is not going to be able to solve those challenges.

Once the AI gets challenged, it's over. The game is over basically, right? It's just going to sit there for like next 40 minutes burning your tokens up. So, we need to basically stop that from happening basically right away. The best way to do this is that AI needs a residential IP address. So, basically, the way IP addresses work is when you're in your house or whatever, you're going to have a residential IP address.

And that IP address usually rotates every once in a while, and it's you know, it's not static, right? So, you have a residential IP address from your internet service provider. In a data center though, if you have a VPS, then the way the IP works is it is static. So it runs on a static IP address. And also it basically like flags that it's data centralized. So if you Google the IP address or you search it up, you do like a reverse local puppet, it'll tell you that it's from a VPS right?

So I use vulture and basically if you look up that IP address, it'll tell you that it's from a data center, which is phenomenal. And that's very annoying because these laughs will look at open me like, nope, you cannot access our website. You have to solve this crazy capture and the AI is like, okay. You know, the AI is getting cooked basically. So we need to basically give the AI a residential IP address. So we do that is we have to have a proxy inside of our house, whether that's through a device or through a router or whatever.

So we need to set up a proxy inside our house. And the second thing that we need is we need a headed version of Chrome. So we need a version of Chrome that's running headed, ideally with a GPU to maximize sort of not getting detected. And we need cloud to basically be able to do that, ideally through an MPCP server, just so it's like normal and stuff. So we need them two things. So the first thing is a residential IP address and the second thing is a fully headed version of Chrome in order to actually achieve any good results with this hack bar, right?

The way I have this set up is the following. So I have obviously a VPS I mentioned. So it's running on vulture. It's where I used to host my labs before I moved to Google Cloud. This is where I used to host my stuff and you know, cloud is running here or whatever. And I have a phone at home. It's my phone. And this phone is a Galaxy S21 and one day I tried to change a battery and the screen got cooked basically. The screen is cooked.

So I basically had to try and save this phone. I managed to type in my password into this phone and I managed to get Samsung Dex Go. So that's like a basically like an operating system on your phone, which is kind of nice. So I was able to actually use the phone and I was able to kind of install a few things. Most importantly, I was able to turn on wireless debugging, which actually allowed me to access the phone through ADB.

And when I was thinking about what I was going to use for this phone for, I was trying to build this AI chatbot, and then I had this perfect idea. I was going to give the AI the Android development bridge, or Android debugging bridge, and it was going to be able to talk to my phone and use Chrome on my phone. And that's basically what I end up doing. So, I basically run the cloud on my VPS, and both of these are connected together on a VPN, so they're connected on the same network.

And basically, I have ADB on my phone, cloud on my VPS, and cloud is able to use ADB to turn on Chrome, it's able to even switch on the phone screen and stuff, and it's able to navigate to whatever it wants to, and I use the Chrome DevTools protocol for the cloud to be able to basically navigate through it with Playwright MCP. And that's the kind of solution I ended up on, and this works quite well. The main reason is that these phones, they tend to pass under the WAFs a lot easier than other devices.

So, even if you have like a Linux computer, that can sometimes get flagged for captures, right? Especially if it doesn't have a GPU, or if it's like running, you know, like crazy stuff like Kali or something, you know. It might get a flagged for basically having like a weird, you know, Linux version of Chrome that is kind of suspicious, you know. When you're running stuff on this like Android, and you're running Chrome on Android, and you have something like Samsung with an Exynos GPU, it's like, you know, as long as you don't jailbreak the phone, you're going to get never going to get flagged because the the chance of a user like that being like a web scraper is literally like 0%.

The chance of a web scraper running on a phone is like 0%, right? So, these WAFs are like never going to catch you on these phones, and that's super nice. So, honestly, if like right now you are thinking about building a hack bot, and I'm I'm going to talk about this in like the last slide of what you should do, but definitely auth is one of the things you need to solve. It doesn't have to be a phone. It just has to usually be a device that you have on your house, like an old laptop or something.

Literally just like install Chrome on it. Like link it with a VPN, expose the dev tools protocol. That's it, problem solved, and you know, happy days. You're honestly better yet, just give Cloud SSH and tell it set it up itself. That's honestly See that the way that works the best is give Cloud SSH so that if it ever does break, which it might, Cloud can just go in and fix it, right? So, there's basically no issue. And that's honestly like the way that I found that works the absolute best for auth.

Now, let's talk about noise. So, noise is something that I mentioned before. There's kind of an influx of reports that look kind of valid, but they're actually not, and they're actually garbage. So, how do we, you know, is there any way to actually consistently get around this noise? Because, you know, at some point you're going to run this hack bot. Especially if you're running it wide, you're going to have like 20 reports in your hack bot.

It's going to take you the whole day to triage that, and you might get like two actual reports out. Is there a way we can reduce the amount of work? Well, there is, and there kind of isn't, right? Let me explain. So, basically, the main idea that I've seen on a lot of podcasts and some like guides and stuff is to build a validator agent that sits in between the agents and you basically reading the reports. So, it will validate the report for you, and it will basically decide whether it's a false positive or not.

The issue I have with this is that these AIs, they like they're basically like either completely in or like hate the report, basically. I don't know what it is, but they're very like emotional. And whenever this validator is tasked with validating a report, it's like a dictator to this AI. Even like hints that that this might not be valid, it just gets thrown out, basically, which is is the behavior the AI exhibits because it's been kind of trained to do that from user data.

And also, the companies have found that the users prefer when the AI is kind of decisive and stuff. So, that the companies are incentivized to make the AIs quite like emotional and quite extreme from one side to the other. And this leads to a situation where the AI will throw out completely valid reports or reports that were like they probably could have been real vulnerabilities, but they were kind of like gadget level.

The AI was like, "Nope." It just gets rid of it, right? So, that's that's what I see a lot. That's why I didn't like the solution a lot. Because I felt like the AI was 100% going to be throwing out good reports. I was pretty convinced of it because when I ask AI to validate certain reports, even ones that I kind of knew were valid, it would like say that like, "Nope, this is not a vulnerability. Nope, this is this is dead." Something like it would be like, "This this is closed." It would use like that that that type of language to me.

I was like, "No, it's not. Listen. Listen, like this this this can still work." And like only after I like I get it to do the whole process, it's going to be like, "Okay, sorry, this actually works." But, you know, if the AI is doing this autonomously, then you're not going to be there to tell that to the AI, and the AI is going to make a decision that's probably going to suck. That's why I decided not to do that, and I decided to kind of pivot to these other two features.

The first thing is I built a scale and kind of like a guideline for what the AI should actually report and what it shouldn't. So, a lot of the times it's going to report stuff like blind SSRF, which I suppose I mean I suppose it is a vulnerability, right? But, also it has like no impact, and I I don't care. So, I decided to just explicitly tell it that, you know, I I don't care about this. This is kind of a gadget. You can record it or whatever, but you know, just don't don't send it to me, right?

It's like I'm not I don't want to look at that. And I basically made the scale. It's called like impact assessment, where it basically just looks at, you know, what the actual greater the report should be based on stuff that I've written it. And this has worked quite well. So, it has thrown out basically the worst of the worst, like the cookie like the the cookie tracker leaks or something, or like the telemetry at CSRF type stuff, which is just terrible.

So, it has gotten rid of like the worst reports, but, you know, obviously, it's not going to let everything through because you can't build a perfect rubric for vulnerabilities, basically, because every vulnerability is different, right? Every vulnerability is going to be different, so we have no idea what next vulnerability is going to be. Next kind of thing was I kind of toyed with this triage agent, where it's different from a validator, where it actually works with me.

So, the idea was that I was going to work with this triage agent, and it was going to basically validating the vulnerabilities as I'm kind of looking at it, and I'm kind of working with the agent to validate and then also escalate the vulnerabilities as kind of a part of my work. And this has actually worked a decent bit well. So, I've kind of only added this recently, and so far it has worked quite well. So, I've kind of been, you know, doing pretty okay with this.

So, the main two things that I've been doing to kind of reduce the level of noise is to give the AI skills to actually, you know, stop putting in bad vulnerabilities in first of all, and then working with the triagers basically giving me a summary of the most actionable reports I should go after right now, and then, you know, start triaging the ones kind of explaining them to me, which has worked quite well. And in terms of, you know, things that you can literally do today.

So, if you're thinking about building a hackbot, this is literally what I would do like today after I finish watching this video. I would first of all figure out auth out, so I would basically find a device right now in my home. I would try and set it up so that it can run a headed version of Chrome, and I would figure out how I can use it for my VPS. Once you have that, that's like the biggest problem already solved.

So, honestly, like don't even build the hackbot if you don't have this. If you don't have auth figured out right now, like a way to get a residential IP address and a headed version of Chrome, there's not even a point in running it right now. Honestly, I'm telling you these web apps are that bad that you're just going to burn tokens for no reason. So, until you have this figured out, do not build the hackbot. Like, I promise you, do not do that.

The second thing you want to do is build your project spec. Obviously, you know, like now, you don't have to code this to yourself, okay? I'm not telling you anything like incredible, all right? AI can build the tool for you, but we have to give it what we actually want. And that's why I always like building a project spec for what the AI should do, which is just like a markdown file. Like literally you just call it project.md or something, and then just put all of the things that you want the hackbot to do in there.

So, I would put that I want this auth to work this specific way. I want this auth to work on the Chrome DevTools protocol. Here's like my credentials for like both VPS and the computer. Set it up that way. I want a dockerized application. I highly recommend you dockerize it. And, you know, I want a PostgreSQL server on it. You know, I want the UI to look like this. So, make sure to be deliberate with it. And I recommend building this document out very well, and then just telling Claude to implement them.

And obviously, you're probably going to have to fix it later. But, other than that, you know, it works quite well. And actually, I think that this is much better than using a service or whatever online because, you know, you can actually tailor it to exactly what it needs, and you can have everything backed up yourself. Third, get a VPS. Honestly, get that as soon as possible. You probably should have it anyway if you have like a VPS.

It's like $20 a month, and it'll save you a lot of a lot of real effort. And I, you know, you you could use a computer at home, I guess, but I think a VPS is just more efficient because one, it's just going to be cheaper, and also just managing it is like less annoying. So, I recommend getting a VPS and using like your computer as like a browser proxy instead. And lastly, just hand the project spec to your AI, build it on the VPS, push it to GitHub, commit any secrets or whatever.

Just easy as that, simple. That's your hackbot done. And you can literally start improving it from there. And after that, you know, it's just going to be iteration because, you know, you're probably not going to nail it first try. I didn't, so maybe with like my knowledge, it'll be a bit easier. But, you're going to have to iterate because it's, you know, a new and expanding field. Just some other tips before I end up the video is you know you want to hold your skills ideally on a cloud database.

You can create your like instance anywhere but you can sync your skills from a database. I'll use Google Cloud Cloud SQL and it's very convenient. It's just a post SQL server and I could literally just like store my skills. I can store like anything I want on it which is very nice. And something that kind of coincides with that is if you use this tool called BBscope like bug bounty scope. It's like a go lang tool and it'll basically pull the scope for a program onto your database.

So this is really great if you have a database you can literally just run this tool and it will pull all the scope from every bug bounty platform including your private programs. If you add in like the odd keys it will pull that onto the post grass database and then it will literally just like let the AI just query the database instead of re-running scans. That makes it much better because it basically you know it it prevents everything going out of scope first.

It prevents you from having to write the scope every time and it prevents the AI from syncing tokens and trying to find out what the scope is. So it's like a triple whammy to just get rid of a lot of problems immediately. So I would highly recommend using that tool. And lastly I would recommend you to write your prompts and skills manually. Maybe this is a bit controversial. I don't know but honestly I think AI is at writing prompts and skills.

Like just you know from my own experience whenever it writes these kind of things it's just like terrible. It's just like it just completely like misguides the AI that's actually reading the skill. It's like really long as well. It will consume so much usage when the AI writes them. And I just don't think it's that great to be honest. So you could use like a I suppose you could like write like a meta skill for writing other skills.

However, I think honestly the cheapest and most effective way to do this is to just do it yourself. Honestly.