How I Use AI For Bug Bounty | Bug Bounty Thoughts transcript

There's been a lot of talk recently about AI in the cybersecurity space from some of the recent HackerOne controversy about them potentially training artificial intelligence models on some of the bug bounty reports that we send in. And from the drop of Cloud Code Security, which managed to find a decent bit of zero-day vulnerabilities in open-source libraries, you can see that the development of AI is rapidly progressing throughout the cybersecurity space.

But what does that actually mean for someone like me, for someone like the average bug bounty hunter? How does AI impact my workflow, and how does it feed into my bug bounty work? >> [music] >> And in this video, I just want to show you some of the things I've been experimenting with with AI, some of the tools I've been trying to take advantage of, and just the overall setup that I've had for my bug bounty work. I'm going to show you basically what I've been doing, going to show you [music] the setup.

I might show you then how it works just on a random website. And then I'll give you my thoughts on what I actually think about this setup as well. We'll start off probably on the number one way I use AI during my workflow as a bug bounty hunter, which is this thing right here. Just to the right you see, it's called Shift AI. >> [music] >> If I can bring it up here, it's this thing right here. So it's Shift Agents. And you can just get this in Caido.

So you can go to the Caido plugin store, basically download this. It's free as well, which is kind of nice. But you have to supply an API key to um to some kind of model provider. So either like OpenAI or like Anthropic or thing I use, which is called Open Router. More on that later, but once you have this all set up, then all you can do is just go into the replay tab, hit this agent button, and you can talk to this agent.

The advantage of using something like this, so Shift Agents, over a web LLM, so something like, you know, chat GPT online or Claude AI, is that when you send messages to it, then it can actually perform actions as well. Say, I don't know, I'll tell it to look at this endpoint, at this endpoint, and add a parameter called URL to it. So, I'll just message this Shift AI here, and it's going to think. And the advantage is that it's kind of built into Caido, that's AI is going to be able to send requests, and then analyze the response.

So, it's going to be able to dynamically change its behavior depending on what the response is, what it comes back with, and it's going to be able to modify and change the payloads we tell it to deploy. That's pretty nice. It's a little bit of a step up from just the normal web LLM, just because it can actually iterate itself, and then it can kind of think as well. It can do a little bit more things, and it has the full context of the request, so it will read exactly what it needs, which means I would say it's a it's a bit better than some of the the web LLMs that you can use.

And then, one of the other features here is that if you press shift control K, you get this thing, and it's kind of like this is kind of like your short little toolbox, I guess. But, like you can type in like something small, like a small little command, like grep for I don't know, whatever, or create a match and replace rule that does this, and it'll actually go and like create your match and replace rule. So, I have like some match and replace rules here.

Uh I think I think that was for like a lab, so you can see here, I created this match and replace rule with like AI, and basically just by just typing it into this box here. And this is pretty cool. So, as the Shift agents, pretty cool way to use AI. I think it's one of the more user-friendly ways to do it, because you're already kind of hacking. You have a request that you want to test on. And if you suspect that there might be a vulnerability, you just shoot it off the AI, you turn the model on, and then you say, you know, try this, this, this, and that, and the model will actually go do it.

Which means that this is pretty good for iterating payloads, which is the main way that I use this. Basically, [snorts] if I want the AI to try loads of different things, I say, try these bypasses, try Unicode normalization, try hitting the AWS metadata, we'll do it through this or whatever. It's pretty good for that. Pretty good for thinking about these payloads. Now, what's it not good at? I feel that it's pretty bad at kind of innovative thinking.

It's that if you tell it to do something, especially in terms of the security space, it's going to start banging its head against the wall and that until they basically give up, right? So, if I tell it to look at this for SSRF, it's basically only going to that, which is Yeah, what I asked it to do, but you know, as like a human, I'd probably try other things after a while as well. And it also only tries things that are kind of common, right?

It's It's not going to come up with payloads that are very innovative or that like are completely new. I find that it's just incapable of doing that. It just basically only recycles payloads that it found online because that's literally what an AI does, right? It's just going to recycle those types of payloads, right? And I find that in that case, it's not very good for finding complex vulnerabilities. Probably good for finding like simple stuff, like if I had like this URL here, and I typed in localhost there, and it popped up and it was like, oh, cool.

Yeah, it's probably good for that. If I needed a complex bypass, where there was some like regex behind it that was validating it, I had to bypass the regex, I wouldn't use AI for that just because I feel like it wouldn't bypass it very well. And I would do that probably manually. Maybe with some help from the AI. Although, at that point, I would probably just use the online like ChatGPT version I would ask it for that instead.

Also, one of the things that can happen is that it can get sidetracked a bit easily. If you give it a general prompt and you point it at this payload, and there's like there's like some specific thing here. Like, for example, I had I had one thing where it I had I had accidentally included um a header I was trying, and it was like an Amazon header. And when the AI saw that, it started immediately testing for AWS metadata, and it kept doing that for like 10 minutes.

And I came back and I realized that it that what what it was doing, and and I had already tested that before. So, I was like, "Oh, for God's sake." And it just wasted like 10 minutes of its time looking for some That's another thing that this AI can do. And the other benefits of this over the other kind of LLM tools that I'm going to show you later is that it's a bit cheaper. But here you can select a model like um the one that I used to use is Rock.

Quite cheap. And honestly, I've tested a lot of these models like the Sonnet models, Gemini, and these Rock models. And most of them just used to do the same thing anyway. So, I I used to just go for the cheapest one. Cuz honestly, I haven't seen much of a difference in terms of the bug bounty work. Yeah, the Sonnet one usually finds the bug faster. When I tested it against some ports figure labs, Sonnet was able to find it faster than like Rock.

Rock was just going in circles, but it found it anyway. But honestly, I don't really care about that either, and it's just Sonnet is just too expensive for me. The costs can rack up very fast. So, this one good thing about this tool is that you can use the cheaper models, and you can save a little bit of money. Now, I'm going to show you how I use Blood Hound, which is one of the newer tools out there, and one that a lot of bug bounty hunters are hopping on.

So, I'm going to show you how what I've kind of experimented with on that and what my setup is well. This here is Cloud Code. It's a website there and I even installed that. I imagine you can follow basic tutorials, I won't show you how to install it. But basically, to set this up you have to either have a Cloud subscription, so you have to have one of these. I think this is also like rate limited after a while, so you have like a fixed number of credits or whatever.

And you can get like these basically like um subscription tiers, or you can set it up like I did using something like OpenRouter. And this is also useful for when you're when you're setting up your Shift AI. Shift AI, you're going to need this OpenRouter API key. And I kind of like this just because the way you use it is you just put a little bit of credits into it. And basically, it's going to use the credits whenever you use the AI model, which I find pretty cool.

And you just kind of have more control over how much you're spending. It's nice because again, for Cloud Code it can get very pricey. I'm just going to show you like one prompt I'm going to ask it to do one thing and you can actually see how much it's going to cost. Anyways, you set this up, you go to your API keys or whatever, you get that API key and then you can just basically put it into Cloud Code and then you can use Anthropic AI models.

So, Cloud Code itself is a terminal like chat is like a terminal command thing. I think you can use it online, but I just use it through a terminal. And I'm going to show you it right Yeah, so I have this like bug bounty directory set up. And basically, what you do is you just install that Cloud thing, you set it up with set it up with your API key from Anthropic and then you can access Cloud through this and you can just press Cloud.

And here it is. Here's Cloud Code and now we can use it. So, we can type in whatever we want to. But, the important thing is to actually connect a few MCP servers onto him. And what this is going to do is it's going to allow Cloud Code to basically do things, right? Because by itself it only it only run shell commands as far as I know, which is pretty useless. We kind of don't want to do that. But, I have installed Playwright, which is a browser like a Puppeteer type browser thing where AI can basically click on things and it can like load JavaScript and it can read the console out from my browser and it can also use my logins and stuff like that.

That's why I wanted to use that. And obviously I have Kaido. This allows it to read like requests. And I specifically use this so that the AI can fetch my scope. And so it's not going to access domains that are outside of the scope and waste time. So, these are the two MCP servers. There's a few more you can use like I know there's like one for Kali Linux as well. And there's a few more like that. So, you can look into MCP servers as well to enhance your Cloud Code.

And then once you have that set up you're here and then you can give it commands based on stuff you want it to do and it'll go do it. Now, fair warning with this. These models are expensive. Type in model here and it should show me a few of them, right? So, these are the models here and the default one is Sonnet. So, it's Sonnet 4.6 the newest one. You can just see for 1 million tokens $22. Whoosh. Decent bit, isn't it?

A million tokens is Honestly, it sounds as like it's a lot, but it's actually not. Trust me. Like it goes down very quickly when you're when you're working with bug bounty. The most expensive one is Opus here with 1 million context. And I haven't tried this before. I haven't tried Opus yet, but it's the most expensive one. So, I imagine it would be pretty good for these tasks. However, yeah, it's going to come with a price tag.

You can You can argue that, you know, if if it's finding you bugs and it's worth the price. But honestly, I haven't had it found I have find any bugs yet, so I'm not really convinced that the ROI is there yet. Anyway, I'm just on the cheapest one right now, which is Haiku, just because you know, I haven't I had I don't I don't really see much return on investment from this at the moment and I honestly haven't seen much difference between any of these models as well, so I'm just using this one for now.

And anyway, basically, that's how you can set up the model. You can set up whatever model you want and you can also change the amount of effort that it's using. So you can make use like low effort or high effort or whatever. So that's maybe one way to actually reduce the price is to use one of the more expensive models and put it on low effort. And honestly, maybe that's better cuz the high effort one is just awful a lot, so it's better to maybe set it as well.

And yeah, the way we use this is once you have your MCP server set up, which is just go online and do that. You go in here and you type in what what whatever you want to do. So say I wanted to find endpoints on the domain and I'm just going to put in my my little bounty domain here. And I also have skills set up, so I'm Cloud Code. This is actually one of the cooler things you can do. You can create skills where you can create like small little documents about specific things you want it to.

So I have a skill set up that tells it how to find interesting endpoints based on some of the methodology that I do. And because of that, when I tell it to find endpoints, it's going to load that skill into its like memory or like brain or so. And then it's going to basically execute the things that I tell it to do there. Sort of it It sort of does and it sort of also doesn't listen to you sometimes, but I'll get more on that in a minute.

Anyway, I'm going to tell it to do that, so hopefully it's going to do something here. And it's going to load the skill. So first it's going to load the skill. Then it's going to open up the browser. So the browser's been opened, which this is Playwright here. And it's looking through a few things here. So it's navigating. And you can see it's going to a lot of places. And I actually never remember telling it to do with that, but I guess it's just going to do itself.

Phenomenon. And it's just going to think. One little minus of this is that it's a little bit time-consuming. This is stuff that takes a little bit of time. Especially Cloud Code, it can take a decent bit of time to buffer on some of these things. Okay, it actually finished very quickly there. Obviously, this is a very simple domain, though. More complicated domains, it'll take a lot longer. It can take 5 to 10 minutes to for it to process.

And yeah, it can be expensive as well. You can see that it found most of the endpoints. I told it to select the ones that are the most interesting. Which is just the default skill configuration I have. If you go here, it's a This is a broken access control potential vulnerability, IDORs, or prototype pollution. Which uh makes sense because these are all my vulnerability labs with actual vulnerabilities in them. So it made sense that it returned those anyway.

But you can see that was pretty good, actually. That's probably better than most of the times that I've used it, actually, which is surprising. Anyways, that's the Cloud Code. And now we're going to see how much that cost. This is the $16.29 I had before. So I'm going to refresh the page. Now you can see that cost that cost um about 60 60 cents for that one prompt of getting Let me see. I'm getting six endpoints back to me.

So I mean, was that worth it? I don't know. I don't honestly don't know if that was worth it for 60 cents on one prompt and you could on on the cheapest model by the way. That was the cheapest model and yeah, honestly that's um like you can see just how expensive this gets. And yeah, it's it's it's it's good though. You can see it got the endpoints back to me and stuff, but it was expensive. Now, when it comes to the results of this, this is more of what I use for like actual vulnerability scanning.

So, I tell it to do some specific task, analyze code, and then give me the results back. One of the big question marks I have on this is how accurate are the results of this, right? We know AI can hallucinate. We know this. But, when this comes back with results, I'm always thinking like is this actually real and it sometimes can take a bit of time for me to confirm the results. Whether this is actually true, you know, I'm looking at this endpoints and it says you know, broken access control vulnerability potential.

You know, the AI does like to exaggerate on some of these security vulnerabilities. I've literally like when on any website, it'll come back with some critical vulnerability every time, but it's basically never real. So, there's a big question mark with using some of these tools and these vulnerability scanners of you know, whether they're actually coming back with with real real um real information. I feel that because we saw for example that one bug bounty program, the Crow bug bounty program, which had just so much AI slop that it had to shut down the amount of AI generated reports that were coming from so things like Open Cloud that were getting automatically submitted that the basically the maintainers that she just shut the program down because of how much AI slop there was.

Which is very unfortunate. But, you can see that, you know, you have to kind of approach the AI with a question mark and you have to, you know, judge the results with your human brain. So, the two things So, the the thing that I I I use this most for is just to do these simple tasks of like identify endpoints, analyze, you know, the source code, you know, look for JavaScript sinks or whatever. And the kind of negatives of this are that it's expensive.

As you can see, it's very expensive. And it's also kind of likes to hallucinate vulnerabilities sometimes. >> [music] >> And you have to take it with a grain of salt. And yeah, those are my thoughts on using AI for bug bounty workflows. If you enjoyed this video, this is more kind of informal style of me just showing you what I've been doing and what I've been experimenting thing with, then leave a comment down below and tell me that you enjoyed it.

Or if you didn't enjoy it, then tell me what I should have done otherwise. >> [music] >> And I would also appreciate that kind of feedback as well. I think at the moment, just to summarize my thoughts, is the AI is getting quite good. You can say that it's getting >> [music] >> it's getting more power to actually do things by itself. Look at requests, you know, visit websites, click on things. And it's starting to act really more like a real human, like a real human bug bounty hunter where it adapts its behavior depending to what things it sees on the website.

It's very good. But it's still not quite there, I think, in two categories, which is one is cost. Just the ROI is, you know, it's quite expensive. And two is that it's just kind of hallucinating still. It's kind of not that innovative. And it kind of just does the same things every single time. And [music] it's not really good for bug bounty because if you do that, you're not going to get bounties because, you know, you have to be innovative.

You have to think outside of the box for that. But I think right now, human bug bounty hunters are still going to stay for >> [music] >> for now anyway. But AI is definitely going to be a nugget more and more in the workflows of a bug bounty hunters. And I definitely encourage you to try out these tools, the ones that I've showed you today. I mean, I've talked about both the pros and cons of them. But, you know, if you use it differently or use it in a way that's, you know, more suited to your style, then you can maybe cancel out some of the some of the cons and get more of the pros.

And, you know, maybe I'm using them wrong. Maybe you can tell me if I'm using some of these AI tools wrong. If you have a way that you use them, then put the comment in the comment down below as well. And let me know about that as well. Anyways, that's it for me. Thanks for watching and happy hunting.