Bug Bounty Singularity (Ep. 181) transcript

I was looking for it everywhere and I'm like this is a dead bug. Who who runs who runs backend JavaScript? You know like you know like those kind of things. A lot of the bugs we find are bugs that I like know that they exist but I thought they were dead. >> [music] >> Best part of acting when you can just, you know, critical think, right? >> Yeah, dude. [music] If you've been in the Bug Bounty recon game for any period of time, you know how beautiful it is when some unsuspecting dev or DevOps guy spins up what's clearly supposed to be an internal facing server and accidentally just gives you the keys to the kingdom, right?

It's a big payday. It's an easy win. It's a beautiful thing, right? And unfortunately for us, Threat Locker also knows about that and that's why they created their zero trust network access product. Okay, check this out. This is a product that prevents that unauthorized network access not only at the network level but also at the device level. Okay. So anybody's connecting to those sensitive internal services, you know that they are authorized as that user but also the device that they're connecting from is validated.

So you get complete introspection into who's accessing these sensitive systems. It's a great product. Check it out at threatleer.com. All right, let's go back to the show. >> Dude, what is up? It has not been that long. When was the episode we did with me and you and Justin? >> It wasn't that long ago, but it was the best, so I wanted to come back on. >> It was awesome. Yeah. Have you gotten any like feedback from it? >> Yeah. >> Reaching out. >> People People liked it.

I hope people liked it. [laughter] >> Yeah. People said they liked it, but maybe they would just say was that when we had demo pop in, too. >> Oh, yeah. I love the demo. We had We had It was like we rub rub a lamp and demo came out and taught us what >> Exactly. Yeah. I I need to ask Justin if he got any feedback on that. That was such a like a unique kind of funny and fun episode. Um, >> you should you should you should definitely do that more like like like rug the demo lamp. >> Yeah, just call random guest to pop in in the middle of episodes.

I don't know if we'll have time for that today, but uh yeah, for anybody who doesn't know uh obviously I'm Reszo Joseph and then uh the guest quote unquote guest uh today is is is excess doctor, also known as JD. Um but yeah, so obviously you've been on before. Um it's not been that long, but we will in the spirit of Justin to make him happy jump straight into a bug. And uh for the topic today, we're going to be talking about um kind of what our hackbot has done.

And I guess this is a little bit of an accountability to force us to put out the blog post that we've been working on for like a month. Uh this episode should release with our blog post on like uh what mine and JD's hackbot has has found uh this year so far. But anyways, yeah, dude. Go jump into one of the bugs that our hackbot found. So basically um that we pointed the hackbot at a particular what looked like uh a GraphQL API.

Um but um that what the hackbot found this was mostly hackbot this this bug. Uh the hackbot found that this was um actually uh in the back end using MongoDB um and it uh parsed um backend JavaScript. So you could use uh the the wear like the um dollar sign wear expression and you could basically run JavaScript and you could like start by just saying like dollar sign u so imagine these are MongoDB works with um uh with JSON so basically you' say like where would be the key and then the value would be let's say one equals 1 and then um you know if that's said true um then um then you know that that this is running backend JavaScript And so um with this you could do this.constructor.constructor um and then you could basically um get do things like uh return global this.process equals process and you you can um get basically get um like like the environmental variables.

Um you could get uh the um access token like which was one of the envir we basically got the environmental v uh variables. Um, so, uh, we did like the access token. >> You've left off the coolest part, though, is that all of this was via like basically like a binary search. It didn't just like respond with the data. >> Yeah. Yeah. So, it was it was blind and it the hackbot completely got that and then we had to reproduce it obviously.

Um, so uh and then so it was a critical uh with a really good critical vulnerability. >> Yeah. So for any listeners, the way that it worked was basically it looks almost like a yeah like a GraphQL query like you were saying but then in nested in the query was some query parameters which were JSON in nature. Yeah. where the key was basically where and then the the like you said the value there um was this doc constructor doc constructor open parenthesy single quote return process env node env zero equals character and um actually let's see if I can show this there we go so basically it looked like this >> down in the down in the body you know this would be considered like one body and then this would consider one body and so if if this matched, then you could actually, you know, slowly binary search out this.

And of course, we had Claude, right? Or Claude itself wrote a script to totally extract these variables. But this totally blew my mind and it was really awesome to see our Hackbot find something so cool like this. >> Yeah. And even in the modern era of of long triage, they triaged it in like 10 minutes. >> That's true. Yeah. which maybe it's because this program doesn't actually get uh that many uh you know reports, but who knows?

I've actually been told by from a lot of people that the triage times been better. Um which I don't know if that's because they're rolling out some of those like signal based uh cues where if you're a you know more of a seasoned hacker, you get faster triage times >> or if the automation kick in. >> But the payment times are still very slow. >> Very slow. >> Yeah. just I was just talking to um a young top hacker um Fab and he was saying that um you know he's had bugs sitting out there for a long time on Microsoft and I and I had previously heard that Microsoft's been you know paying well so >> I love Fab that guy's great hacker >> he is okay yeah so um yeah I can talk about another bug um and I may have mentioned this one to just on the pod but I I couldn't remember but there was a Google product um it was like one of their you know less high severity services like one of their other services, but um it was responding with an uh OTP like hash like a brypt hash of a OTP in the response when you requested an OTP and um it was actually crackable and you and I reported this. our hackbot found it basically manually.

Um, and it it was like I don't know if it was like a specific type of decrypt hash, but anyways, it said it would be crackable in like, you know, two hours on rentable viable hardware. So, you could so you could basically just, you know, reset someone's password and then crack the OTP and then take it over like that. Um, so that was that was pretty sweet. But all right, we got a bunch of sections in here. JD, you've you've like kind of broken this down in a cool fun way.

Um so this first section is integrating AI and manual hacking workflows. So yeah, so like I I guess um I guess what I was getting at there is you know when I used to only do manual hacking um and um and then in the you know in the last few in the last year or so I I I'm basically hybrid like I do a lot of manual hacking and I do a lot of AI hacking and I I I usually use AI in basically everything that I do even manually.

And I think that the um the way that I I like to you know the how I've transitioned to do that is you know I first of all I don't have much time in life to um to just hack straight because I have I have another job and I have kids. Um and so like there >> literally a cardiologist with a family and kids. >> Yeah. And so like you know when I like when I used to start manually hacking there was a lot of friction that I'd have to overcome and I would often give up on programs because I' like I just I can't I'm looking at it like Google like I'm looking at the protobuff I'm like I can't I just don't have the emotional energy to do this right now or like I would look at a at a at a program that that was it had a lot of that looked like the the threat model was um was very off related like you know there were multiple um there were multiple um privilege uh privilege types of users and you can create a million users like the archangel types of bugs that are great and pay really well and like you know uh you know really um are great bugs and if you want to actually hack on a program you have to do that but I I couldn't do it like I just my my emotions I like I'm like why am I doing this I should be with my kids right now like why am I making technician and assistant and testing every single endpoint against both of them >> but you know AI has really helped me do that you know uh like basic.

So the first thing that it helps me do is is the things that I can't do alone. Uh and I'll I'll set it to do that. Now maybe it doesn't do as good a job as as Archangel. Of course, I know it's uh it's still AI and not and not a top hacker, but it does a better job than I was doing at those bugs. Mhm. >> And then the next thing is that um when I first approach a a target, I get very anxious when you when I first look at it.

Like I look at the I you know I I walk the site, I'm like, "Wow, there are just so many features here. There are so many there's so much JavaScript. There's so many chunks to look at and and that um like I don't understand the app. It takes you like days to understand an app or longer sometimes." And um so I I will use AI every time I hack to do like a once over on the app and to do OSENT on the app >> and then to write me like a book >> on the app and I'll like in my off time when I'm not which I don't have much of but like let's say like if I have 30 minutes or whatever if I'm waiting for something then I'll start I'll read their assessment of the app and then I still won't understand it right because it's just me reading a But it I will understand it much better than than I would have originally just going into the app. when you do that question because I feel like I often >> uh struggle with >> how to best consume that sort of stuff like I think everyone listening probably is using AI to do stuff and it'll often you know in cloud code like all of the information or in codeex is kind of streaming by and so I know a lot of people obviously use like other harnesses or their you know their agents are like routing stuff to disk whenever you're saying like write you a book like do you have it write markdown and then you just read NVS code like what like what are you typically doing? >> Yes, that is exactly what I do.

I write it. I have it write markdown. I have honestly I have AI only. That's how I write have it write everything in markdown in MD files in in VS and I read it in VS Code. >> Do you read the first rendered or do you read it unrendered? >> I read it unrendered. [laughter] I read it unren. I'm just laziness that I don't like. Anyway, >> I can never remember the keyboard shortcut. It's like command shiftp or something. >> I didn't even know there was a keyboard.

I didn't even know >> there is in VS Code. There's a single keyboard shortcut that will like open up the rendered markdown. Oh man, I'm gonna do that from now on. That's amazing. >> So, so I read it and I even when I'm reading it, I am not I'm not intaking it as very well, but then I'll go back to it multiple times during the engagement. And then especially it's helpful after I find all the bugs I think I can find. Like I'll like I'll like find all these bugs.

I'll put them in the folder. As I'm finding the bugs, I put the reports in a report folder in my like main folder. And then like there's also a folder called like initial or like recon. I I name it random stuff. I should have a better naming system. And then at the end when I've when I'm like I can't find any more bugs. I'm done finding bugs in this app. I'll ask the AI to look at the bugs, redo the book, and then I'll read the book.

And at that time, the book means a lot to me. at this time that time I understand everything about the book and I'm like wow like there's there's oh wow I forgot about that about that API there's a there's you know oh wow that that's important there's like a um a dev endpoint there I didn't know about that or a dev um domain I had no idea there was that let's and then I go back and I always find more bugs after I do that >> yeah it's a good idea I feel like um I just end up with like so many of those markdown files all over the place like I feel like I need to come up with a better method for like you you know, telling the top kind of telling codeex or cloud code like always do it like this.

Um, >> my folder is a mess. The folder's a mess. >> There. And sometimes it won't save the thing in the special folder. It'll it'll be on the the base folder. I have a I have a folder called bugs. I have a in my on my desktop and then like I have little folders within that, but like if you look in bugs, there's like random bug reports because >> Yes. because it saves it in the in the wrong. >> Even my home directory is just like so messy, which obviously, you know, you can just have a organize it later, which that's like a little pro tip.

If no one's done that, it's really fun to have like cloud code organize your downloads folder into like like something that makes sense like it's by types and by other stuff. But >> although I once had AI give me a message saying, "Less learned. Put into my memory. Do not RMRF before reading >> the folder." I'm like, "What did you just do?" >> So, what did you do? Yeah. >> Not yet. >> Um, yeah. So, I mean, basically, you know, the whole premise of getting you on here and us like kind of co-releasing this with the blog post was, and I think this would have been much more groundbreaking if we had like posted in April or something.

It feels like now literally everyone I know h has like massive awesome hackbots which basically just to tell the story of us kind of um you know getting into building the hackbot that we built and then some like you know pros and cons and you know some of this has been discussed on that last episode with uh me and Brandon and Justin. Did you listen to that one about like the >> Of course I did. Bro I listen to every episode everything like I'm done. >> The motivation one I feel like was pretty interesting.

I have a lot to say about that episode actually because I have there are like like I felt a lot of that uh a lot of the the onwei uh you know as a lot of a lot of the that like >> not not having passion as much passion for bugs you find with AI which is true >> but like right now I'm doing this thing that is like um >> I have to do something with hardware for this thing I'm doing. >> Yeah. Yeah. And um and it's like gi it's like revitalized my like passion for hacking.

Like I wake up like excited to hack because it's a new thing, >> you know, and I'm learning it myself and and AI is certainly helping me learn learn it. >> Oh yeah, we got that section up above. Yeah. Like just one amazing way to use AI is for just like learning new stuff, right? >> Oh my god. I when I I have to like with this hardware stuff like I I um I literally I'm like I don't know how to do this. I' I've never done this before.

And I'm like, Claude, how do I please like write me a study guide? And I'm following it. >> Um, you know, so yeah, >> one thing that I get in the weeds with whenever I'm using Claude code is like reading all of the things it's doing between its replies. And then even when there is a reply, it feels like the stuff above that's messy. I wonder if I if I should experiment going back to using like the desktop app for some stuff like obviously the code section of it so it still has access to disk and all that stuff but I wonder if it would just hide that noise that I feel like sometimes makes it feel a little overwhelming to be using you know >> if I'm not in the terminal on my computer I am sad >> yeah that's true in general but I'm just thinking like as I was listening to what you were talking about like >> I was imagining myself like asking Claude like you know describe to me this build me a build me uh um uh what's that called?

Build me like a little mini app to like teach me how to do something, right? Like some sort of like let's say new front-end vulnerability like teach me about CSPT, right? And I feel like as it's talking to me, I'm like imagining myself looking at the terminal and then just trying and then also like wanting to understand and like reading all the stuff above that, you know, all the stuff it just built and all the stuff it designed.

And I feel like that makes it uh feel a little bit more intimidating. >> Yeah. No, a lot of the a lot of the time it prints a million things and I and I can't and I'm like, "Oh, yeah. I'm not going to I'm just going to like uh All right, do it. Yeah, exactly. Yeah. >> Yeah. And sometimes it gets annoyed with me because it knows I didn't read what it said. >> Mhm. >> As I said in the previous sentence, >> that's funny.

Like when it starts off thing, it's like as I just told you, you idiot. >> But like the the truth is like whatever AI does, like if you're good at the thing already, >> it's going to do better than it would have if you weren't good at the thing. like if you like like um and so getting good at things using AI will make your AI be better. >> Yeah. So let's get into like basically the story. I think that that's like kind of interesting to you and also you've taken the time to write a huge part of the blog post about how we kind of got into building the hackbot in general.

[snorts] Um you know I think that you know it can be boiled down to like actually did I reach out to you or did you reach out to me? Uh we were talking a lot. >> Okay. >> Uh we were talking a lot randomly like you know we were hacking to co-hacking on different topics and talking about AI a lot. >> Um and then you like came to me and you're like listen >> the hackbot thing is going to happen. >> Like the hackbot apocalypse is going to happen.

Like you and I need to get involved. That's what you said basically. You're like we you and I need to do it. Like if we don't do it like everyone's going to have a hackbot and we're not going to have a hackbot. and we do a lot of AI together and so let's do it together. That's how you framed it. >> Yeah. Yeah, I think that's right. Yeah. We were basically sharing skills already like just you know improving >> and we were finding the same bugs.

We when we were hacking together we were finding the same bugs at the same time like let's just combine. [snorts] >> Yeah. So that was back in like January like really shortly after Opus came out. Um, and yeah, I mean, obviously I think there are plenty of people who have scaled their hack bots just as well as us or better, but um, JD and I have found like what would you say like 60 to 80 good bugs, like solid bugs, way more than we would have done in the past, you know, separately or even together. >> Um, and I think we've had just more free time to spend on other things as well.

Like I don't think personally I've put in as many hours as I did last year, which part of that's just I don't think I would have been able to based on like the family demands and and trips and interruptions and all the things. But um and so that's been awesome to be honest. I do think that like the benefits of it are also huge. Like I don't think not only are we making more money and finding more bugs, but I think legitimately we are providing a much better service than I ever did before.

[snorts] >> Do you feel that way? like when you do a pen test or when you do a >> Oh, yeah. I've started doing some pent tests here and there and I'm I'm like I find a lot of good stuff. >> Um and and whereas I don't think I would have found uh that good stuff if it wasn't for AI some amount of AI automation, >> right? >> Um and then as far as singularity goes, I would I wouldn't of like the 60 bugs we found, I I probably would have found like six of them, >> right?

In the time you put into Oh yeah, maybe less because especially because a lot of the bugs it finds are not the bugs that I would ever look for. >> Um like that that one that we just spoke about. Um I like I told you I used to look for that because I took the Portswiger Academy no no SQL uh uh whatever module and then I I was looking for it everywhere and I'm like this is a dead bug. Who who runs who runs backend JavaScript?

You know like you know like those kind of things. A lot of the bugs we find are bugs that I like know that they exist, but I thought they were dead. >> I because I just I looked for them so often and I never even saw like a hint of them. They weren't even using that technology. >> They exist on one in 10,000 hosts. And so it's like it's not even worth checking for them, but obviously hackbots can check for it everywhere. >> Yes.

And that's like I think that's such a takeaway from the hackbot. Like like like let like offload your stress to the hackbot. >> You know what I mean? That's like the That's like such a That's what I try to do. I try to offload my stress to the bot and then I do the things that I enjoy doing. >> Yeah. This makes me just like think about how the places where we're most useful are the places where the hackbot and like all of these hacking agents struggle and you know you could move it across the line and then that makes me think like bringing good we talk about leads all the time and gadgets all the time but like if there were some way to build in or figure out how to make these coding agents or hacking agents get a good um Spidey sense and not for actually finding the bug, but then when to hand it off to the human.

That'd be like really interesting. Like could you imagine if like through the last six months, not only did it find and report bugs, but if and like it's always finding like leads and gadgets, but [snorts] like there's too many to really go through and like vet all of them. But wouldn't it be pretty cool if like over the course of the last six months it had delivered like you personally like a hundred like JD specific leads for things that it like >> thinks are vulnerable that you could get across the line.

We should build that in. The only problem is in my I watch I watch the thought process constantly. I'm always watching the thought process and >> it has terrible spidey sense. >> I know it has really bad spidey sense. Like it like it like it knows when a bug exists but it think it it's like oh >> that's dead. That's that doesn't work. And you're like no that was that was a you were rate limited right there. you know, like that's not uh that's not and there's stuff like that or the opposite happens too where it goes after something that is obviously not vulnerable because it sees some weird header thing that it that was that I would have never, you know, but then again sometimes the weird header thing pays off for it. >> But I I don't think it has a good spidey sense. >> But what is good is is the gadget finding >> and I also think there's like two different like universes of the hackbot, right?

There's like the what we what we did is we did a a wide scope hackbot, you know, and and like our wide scope like we we basically pointed it at everything and and gave it our our skills that we that we honed very well >> and but there's like the um the um the other kind of hackbot like from the other a few weeks ago the Google episode. >> Yeah. >> Um which is the deep hackbot, right? one target and you know everything about you already know everything about the target which seems to even work better because I you know he he did great >> yeah um but it's nice because I usually go deep and not wide so it's nice having another thing that I have that goes wide >> yeah that's true >> yeah it makes me think we should build a hackbot making hackbot we should build something that builds a [laughter] whole bunch of the deep bots and then runs them >> we should make a hackbot that makes one of those we should make a hack hackbot that makes a hackbot that makes a hackbot. >> Yeah, sure.

[laughter] >> Then then we'll [clears throat] be even better. >> There you go. Uh yeah. So, let's talk about these like negatives that you wrote down here. >> I basic Yeah, like it can suck out can suck your interest away. >> Yeah. I honestly think that the onwei that you were talking about, which by the way, if there's any non-native speakers, onwei is a very actually it's probably like a French word or something, but >> it is a French.

It is a French it means restlessness resulting from boredom. That is the definition of a >> Wow, look at you, Mr. Wikipedia over here. >> Ah, that's what that's what happens when you're 43 years old. You know, stuff like that. >> Yeah, I do think that there is something about an like a, you know, a fake intelligence, finding bugs that you would struggle to find or that you would never find that does impact the soul in some way.

It does impact the motivation in some way. Um, obviously it's cool and it's great if you can get credit for it and make some money off of it. Um, but it definitely I think kind of just like slowly wears on you. Um, I do think your second point here that we're kind of back to being automators is pretty interesting. Um, actually, did you see Hackloop posted a blog like an hour ago? Did you see it yet? >> No. >> Uh, it's it's like exactly this topic that we're discussing.

It's really interesting. Basically, it's the whole is Bug Bunny dead again conversation. But in it, the analogy that he uses is that it's kind of like watch makers. So originally when watches were being made, they were like an artisal craft and they were like very difficult to make and it was like only high-skilled people who could do it and only the best watches actually kept time. So people always wanted the best highest quality thing.

And then there was something that came out that was basically called a quartz watch and it's like you know quartz I think it's quartz um like keeps time perfectly and it allows you like mass-produce really cheap watches and uh there were some companies that that killed but then there were companies that adapted and started making quartz watches and they started thriving and his whole point in all this uh and his whole blog post is very long and all written by hand so you should go read it but um is just that the bug hunters that you know pivot into using AI effectively to find deeper, cooler bugs or more bugs at scale are going to thrive and other people won't.

Um, >> I agree. I think I may agree with that. I don't know though, but we we've done well. I we've done better than I thought uh I could do. Uh, you know, >> I think this discussion it comes down to do you believe that this state that we're in now will continue? If so, then of course Bug Bounty's not over and it's going to keep being in this golden era and thriving or >> That's sort of a different Yeah, that's sort of a different is Bug Bounty dead.

I don't even know if I want to dip my toe into that. >> Well, my point my point is just that like you're right. Like right now it's awesome and the the the bifurcation is just does it get worse as all the bugs are found or do we just keep going and there's tons more bugs and we keep finding stuff? Who knows? >> I think it's it's just like anything is how you use it. Like I like if like right now I'm on I'm doing this engagement and >> and I'm I'm doing a lot of manual hacking and I'm using a lot of AI and it is just going great.

Um and like much better than I would have done with the same skills without the AI. >> Yeah. >> Um for this reason because but you have to like understand what it is and you and I think you have to not stop >> doing what you do like you know what I mean like you know you can't be like okay I'll just let AI do everything like it and and I'm liking that a lot uh too. Mhm. >> Um but the on the onw wee is real and the and um and like when we found that bug that we talked about at the beginning that was great and I know we're going to get paid really well for it but like finding like a medium by myself even feels better than that but popping. >> Yeah.

Uh, and he does talk about that in the in the blog post. Uh, hackloop does just to not to um overemphasize that blog post, but he he mentions the fact that like a critical like five years ago, you would be like really proud of it and it'd be like pretty rare. And now it's like, wow, there's just like so many highs and criticals being found all the time, both by yourself and by our hacking agents that it does feel like less good. >> That's true.

But on the on the flip side to that, um if you like sometimes when I find something like that, not not that particular one, but like some bugs that I really had no I had no idea that this type of bug existed. >> Yeah. >> I'll I'll replicate it like to learn it. >> That's true. >> And I would have not and like another thing that's interesting that I found from the hackbot >> is like I spent all of last year learning client side.

Mhm. >> Like that's like what I what I learned and I I always said to people like I don't even understand what being good at server side means because like client side is like kind of a skill that you can learn. You understand the window references and stuff but like server side like what when people say good at server side what does that even mean? But now I'm like seeing all of the bugs that are out there on the internet. >> Yeah. >> Like I like these bugs are coming in and I'm like oh wow like that that exists uh here and probably elsewhere.

Yeah. So like I'm I'm kind of getting better at server side watching these bugs come in and reproducing them and understanding them. So that's something >> I feel like server side I feel like server side is just all like regular expression uh bypasses. [laughter] >> No, I think server side is like is like knowing what could be a bug on the server side and then looking for it. >> I don't know. I mean that you know what I mean?

And like I I feel like I didn't like when you when you do stuff like the training when you do like bug bounty training and and classes and port swigger and then you go back on into the real into reality and you don't find those any of those bugs. Um it's very demoralizing and you're like oh like oh I just learned all that for nothing like what like what is happening? I'm not finding anything. And then um that's how I felt at least.

And then um I sort of got into client side and I started finding stuff and I I understood that. But like I still I like veered away from server side because I'm like I don't I like that stuff doesn't exist. I'm in my head. That's what I'm thinking, >> you know? But now I'm like sort of seeing what actually exists so so I can look for it. And I'm and I'm finding more serverside stuff myself >> as a result of this. >> Yeah.

In my head like I never thought about this. It's almost like if you imagine like a huge uh X and Y table where there's a whole bunch of cells like you know let's say it's a 10 x 10 there's like a 100 cells like learning hacking techniques both server side and client side is like you're kind of like filling in all those cells >> like those are like the things that you're like learning and then like let's say that you have like full coverage let's say you know all the bugs and so they're all green then like when you come to a new target you have to like check for all those bugs across all the scope and then the and like so you can imagine it's like another you know 10 x10 grid and [snorts] then if you think about like the bugs actually exist is like a third 10 x10 grid that's like all green but there's like a few red dots there's like a few red cells and those are the actual bugs and you basically have to kind of like check every box to overlay it to really figure out where the bugs are >> and you have to understand like what kind of bugs would exist on a particular type of target and that's what you look for like that I guess that's what being good at server side is anyway this is deviating, but that's um it I thought about this a lot because I want to be good at at everything.

I want to be a good all-around hacker. >> Well, dude, I think that the the push into hardware stuff is really cool and really high value and I think will pay huge dividends long term. >> I love learning a new skill. Like when I was learning JavaScript, that was like the best. Like I I like love that. So I was just walking around in a good mood all the time and I'm and I'm kind of like that right now when I'm like about to learn something new. >> Yeah.

Um, and AI I couldn't have done it without I couldn't be I haven't done I couldn't be doing it without AI and we'll see how it works out. >> Yeah. [snorts] All right, dude. Um, let's hit up these last couple sections which is how we did it. >> Do you want to talk? Yeah. You want to talk about the thing? Let's let's talk about how like the thing. >> Sure. I don't care. Yeah. So, and also you mentioned Singularity earlier and we never really defined it.

Our hackbot is called Singularity and that's after the the idea that I had that one day there would be like >> Huh. >> You did that. You you named it that. That was a good name. >> I appreciate that. Yeah. I just thought that you know and I mentioned this in my TEDex talk that I did like two and a half years ago or whatever that there would be a point that's almost like the singularity but it would be called you know the bug bounty singularity where there would be a point at which token cost was cheaper than the output of the of the findings that it would come up with and then at that point everyone would just scale up and I think everyone is kind of doing that.

I think what prevents people from just spinning up a hundred cloud code instances is that like you still have to do manual validation on a lot of them. And so you do end up at a scale where like you literally can't um and and you have to like keep off alive. We'll talk about this in a minute, but you have to like >> you have to keep the servers alive and you have to keep the O alive and you have to like keep improving your system and so all those things have like a manual labor tax. >> So there's like multiple parts, right?

So there's like the the actual thing hacking the the the the program. There is the thing which tells the AI to hack the program and which program to hack and how it hacks. And then there's us interacting with it somehow, >> right? Like that's how that's how we have to think that's how we thought about it, right? Like in the beginning we're like how do we do it? Like we have to inter we have to >> this thing has to hack.

We have to somehow make that thing hack and then we have to somehow interact with both the output of the thing and hopefully even into the thing. >> Yeah. >> Right. And then you came up with the idea of a discord of discord for the inter for both. >> Yeah. I mean people were actually I think this is before uh openclaw but you know I we just wanted to be able to control it from like the stuff I was doing. So before we even had the hackbot, I was like interacting with my um Claude code instances for just like development purposes and hacking purposes via Discord.

And so yeah, we set it up as like a custom Discord for the two of us. And it was Justin originally, but Justin really never had time to hop in. But >> Justin's like, >> "Yeah, >> I'm not doing that." >> Um but uh yeah, so that's how we interacted with it. And like you said, you know, we that second mode obviously there's the hacker, but then there's al which some people call workers or whatever, but then there's also like the boss which is like in charge of like kicking those off, delegating things, telling it to go back and hack harder.

Um, and you know, you can like include or exclude different types of things like does your boss also choose scope or does that come automatically? Does your does your boss also figure out O? Does it also take notes? you know, like or does your does your worker take notes or do you have an overseer that then takes notes, you know, and then there's like but a lot of different systems for how you're going to log all of the output and how do you pick back up from a strand if you really need to or want to. >> I think like each of these three parts caused trouble for us at a lot of points in the last how long have we been doing this now?

At six months. >> Six months. Yeah. >> Yeah. Six months. Um and like like for example like the the um the let's say uh the first step is getting the thing to hack the target like there are multiple ways we could do that right we could do that on computers we could do that in the cloud we could do it one at a time we could do it 10 at a time and uh when do you run out of tokens is an you know like how many do you spin up to run out of tokens when do you scale up the the operation these are all like like hard questions that we had to answer I Guess this episode is for people who want to do this like so like all of uh like all of these are very hard questions and it's it's all trial and error. >> Yeah. >> Um and then as far as the interacting with with it like it sending information back to us, we went through multiple iterations of that. >> We went through a dashboard phase.

We went through the Discord bot which we're still we're still on. Uh we thought about um we thought I don't know we thought about a million things like how to interact with our phone how to get get our computers there. >> I think finding still outputs as like PDF in the findings channel but then we have like markdown in the reports channel. >> Yeah we do and we have mo you know it's kind we have a lot of ways to interact with it but like thinking about it these things are going to pull back to us like multiple true and false positives all the time.

Like there's like every day how many how many alerts do we get from it? too many >> 50 alerts, right? And like I look through them a lot, you know, constantly and so do you. and we try to find the ones and I get like a little XSS doctor exclamation point if they're good too from >> um and then but like sometimes there's stuff that is like kind of lost in that mix >> that may be a good gadget for >> for another day and we have to kind of and we've gone through iterations of that too like >> um of how to of how to uh parse that.

Um, and then um then there's the how do we interact with it uh part of the story which um like at first we couldn't interact with it >> and then we made little logs that we could click on and type uh to it to inject into the bot. Um which I think was a very good idea also of yours. >> We don't really use it anymore but yeah >> I use it I use it like crazy. >> Yeah, of course I use it like crazy. Yeah. And I and I I changed stuff based on it.

That was our big that was like a big change when we were getting all this crazy. So So at one point we were getting not that many findings, but the tokens were being used up constantly like like within two days. And so Reszo's like, "Dude, you got we got to we got to see what's happening. Like what is happening?" And he like ran all these metrics. It turns out like 80% of the time it was using 80% of tokens on off authorization.

So it was trying to log in over and over to the application like multiple ways for 80% of its tokens and then it was only hacking for 20% of its tokens. >> And we found that out looking that that and then we would like look at the log output and it' be like >> let's try to solve this capture a 100red million times. >> Right. [laughter] >> And so we had to that's something you had to kind of deal with here. >> Yeah. I'm surprised that you still go in and like uh inject into the live running log sometimes.

Uh to do that's what I do. >> That's awesome. Yeah. The when we abstracted it out basically we used to have like a thread per >> like a thread per every instance of every worker. Yeah. But now it's like the boss and so you don't actually see like the actual >> it's like a good the good thing we could actually change back try to add back. >> Talk about that then. That's something else we did that was interesting. >> Yeah.

Yeah, I mean just in general, um, we used to have like these hard-coded phases where it would go from like, you know, recon to then hacking to then like looking everything over and trying to chain together bugs. Um, but, you know, I feel like you get a lot higher accuracy, a lot better output if um, the like hacker worker, the hunter worker is [snorts] told like, "Oh, no, you missed stuff. go back and work harder or go back and and some people I think I think uh Justin and maybe J or and um and Douglas DJ [snorts] both um have have that hardcoded where it just says like you're doing great keep going or no no try harder um whereas we we have an actual like kind of like boss you know or overseer that's no it's like oh no you missed this and this and go back and do that >> and um it's hard to know if that's increased accuracy like I feel like our bug throughput stayed the same increased accuracy >> oh did definitely increase accuracy.

Yeah, dude. That when like that idea that again you came up with I every time every time I'm you're the big you're the idea guy. I'll I'll tell you that all these ideas yours. You're like let's let's put an LLM in between us and the worker of some sort and that our findings like tripled after that. >> Really? I feel like they've mostly stayed constant but I but I do think they were like kind of like u what's it called?

Like they were like stalling out on us and then we did that picked back up. >> That's what it was. >> Yeah. >> Yeah. they were throwing up. But that like LLM in the middle was is was such a good idea. >> I appreciate that. Yeah, we got a All of this is like I feel like I've been in a little bit more of like a a Debbie Downer headsp space after coming back from the most recent vacation and I don't know why. Um so anyways, um but I feel like it's affecting my my view on that.

But uh yeah, so [laughter] obviously uh we just like everyone we have a lot of false positives. I actually think that you know that's something that has been discussed a lot on the pod over the last six months. but is still something that is um a huge factor at play for why this hasn't scaled even bigger because obviously like people have to like vet their own false positives and you can kind of bake it in a little bit um like you can you know of course bake in like you know ignore cores issues or most of the time cor issues aren't this or whatever um but I do think that like uh there can't be like full takeoff on everyone just running a million hackbots and just autosubmitting things and I think there are probably very few people who are like scripting the submission of bugs.

One, [snorts] because it's nice to read the report, but two, because there are still going to be like some false positives in nearly any system. >> 100%. I I uh now I mean, I had some I had I had some I I in the beginning of this I like trusted it too much and I and I had some bad situations, but um that the now what I do is first of all, I always now personally like make the thing do run the PC. I have it right every time uh I get something I have it make me a PC and I make the PC proxy through >> Kaido. >> I like every single time.

And then I'll run the PC. I'll look at each request in Kaido and then I'll very clearly look at the scope because it loves to go out of scope. >> Yeah. um not necessarily out of scope, but not in scope, you know, like there like um it won't it won't particularly go to out of scope if if you if it's put out of scope, but it it does not care so much about that in scope part. >> Um and so >> especially if it's referenced from inscope, which you know, as bug hunters, we often report those too.

But if it's like a reference or an API call from an inscope asset, it'll like happily go there. Yeah. But like a lot of the time when the reason why the bug exists is because it's in one of these programs that they only care about the inscope scope and it's not in the inscope scope and and and then that's why it exists because no you know >> Yeah. No one's looking at >> Yes. That's that's a and I don't really care about false positives so much anymore.

I care about at least if there >> because false positives tell you something some a lot of the time like you know if you look closely but I as long as there's like at least 20% true positives >> I'm fine with that I can look through some false positive reports. >> Yeah, of course. I mean honestly it it is like one of those things where it's like uh running it is just like also fun because like we do like eat and breathe bugs.

It's just like fun to see like potential bugs. Like I like I just like looking through the output, you know what I mean? Yeah, I love looking through the output. >> Um, but yeah, but like other we've got we've had a lot of there have been a lot of uh I think the hardest part for us was Oth and it's still the hardest part. Yeah, to be clear to everyone because anytime we say O, I also have this like moment of like wait there are like kind of two forms of O.

Like one is your agent O like how are you how are you authenticating to >> trouble with that >> claude or codeex right and we've talked about that a long time ago but and then there's also like O to the programs and the second one is really you know still the holy grail and when you can solve that really well you can find way more bugs. It's the hardest part is that um like if we think this is a a terminal application and if we if you use headless terminals that limits you in terms of um the way that interacts with with with websites like a lot of the big websites use like capture obviously and it's very hard to beat a capture in playright um no matter what and there's like people who sell you things that say it can do it but Like if you think about it, like look at a look at a um program like Amazon.

Like Amazon, they're not making that capture to prevent hackers from getting in. They're making the capture to prevent people from from um uh scraping the site >> and stealing Amazon. And like they have an entire like building, I don't know if they do, but they have like a lot of people in Amazon whose whose purpose it is to prevent you from scraping that website. And if you want to get an AI to log into Amazon and >> do anything, it's it's extremely difficult headless. >> So you have to do it with headed which is which is possible but then you need an actual computer >> with a screen. >> Um which is fine but that gets expensive and like you going to use you know you're just going to have a computer sitting there >> and that also has problems.

We had that too but then it kept restarting. >> [laughter] >> it kept updating like stuff like that would happen. >> Um and then >> we still haven't solved a lot of this guys like we're not some like super experts um who have solved all this. And then you have other programs that um like o reset like like it it refreshes the O token every 15 minutes and if you log in anywhere else um it it it you know it that refresh uh cancels out invalidates the refresh on the on the thing >> and then sometimes you'll have solved it and then like 3 days later >> because if you you're continually refreshing the token every like 10 minutes but then something changes on the server side or something happens and you lose off and you look back at this thing that you made to keep off and you're like, "Oh, has it worked in like 3 weeks?" Like, "No wonder we haven't found a bug on this program in three weeks." >> Uh, so it's very it's a very hard um that's we that's that's why he did so well.

Um Brutecat did so well. That's not why I he did so well for a lot of reasons. That's one of the reasons it seemed like he did so well um on Google because he he took that out of the equation. He did. He did. He just like solved author was always there >> which was genius. >> Yeah. >> Yeah. Because the whole world is is looking for unauthenticated >> like like there are people that are automators who like are even better without AI than we are with AI at finding unauthenticated bugs.

But like that authenticated realm is where the bugs live. >> Yep. All right. We've got just like a couple minutes left here. I love that you have the successes section. I feel like it's honestly where so much value lives for listeners. And so we'll kind of run through this even though this is a little bit of the secret sauce. One is the validator. Um, and actually I'll go ahead and mention also the ex escalation agent. >> Also both your ideas.

I'm just going to I'm just popping that in there in case someone in case someone thought it was my idea. >> Yeah. So uh we implemented an escalation agent that like when a finding comes in before it goes to the validator uh which validates the bug and writes the report it attempts to escalate it. And so we actually have seen a lot of success out of that. It's really cool. We have these like escalation logs and it'll be like oh escalate this low to a medium or this medium to a high or this medium to a critical.

Um and uh then the valid the validator um is kind of similar to that in the sense that it's looking at a finding. Um sometimes it's already been escalated and then it's basically told hey you need to replicate this in like a kind of skeptical way as if you were a triager for the program and you know if it if it totally checks out and validates then you have to write the report and then it gets all of the report writing skill and all of that.

Um, and so I think that those two have been absolutely huge and I think everyone kind of needs those in their hackbot. Um, did you want to say anything about those before I went down the list? >> Like I want to say like it's again it's another adding un surprisingly to me adding more LLMs into the chain, >> right, >> is better. it it for some reason it ends up being better than adding then when you you'd think it would be fewer LLMs and more people >> but I think you if you add more LLMs you they don't filter out they they they uh they they hone the thing >> you know like the validator it's very I look at the at the unvalidated bugs by the way I I look at them and uh and it's very rare that they're wrong >> that it's wrong >> you know it's just like it and I think the idea is like it gets excited, you know, it wants to find us a bug and then it's like, I found a bug and then validator it wants to not it wants to invalidate.

You know what I mean? >> Exactly. Yeah. Yeah. >> Anyway, that's >> Yeah, you're fine. I don't want to keep you too long because I know we're kind of at time here. Um, one of the other cool successes is Hail Mary mode. um which is basically where we just throw everything at the wall and say, "Hey, this I don't care if you can't find a bug in this, then you need to find it in then the libraries in the in the dependencies.

And if you can't find a bug in that, then I want you to find it in the protocol itself. It's just like you basically just tell it to go balls to the wall until it finds something." Um, and that's found a couple things. It hasn't been the best. I still think there's a lot of like running that Yeah. Yeah. There's a lot >> every time we run it, it finds something. every time we run it and find something and and we've submitted those things.

I just don't run it enough. >> Yeah. And the best part of it is like is that the the the the worker guys tell the orchestrator, the Hail Mary orchestrator, like the orchestrator is like, "Hey, go out and fuzz this this thing, you know, and like find everything you can and think about the fuzzing and and do do weird mutations on the fuzzing." And we have a whole agent who does that and then gives back the fuzzing list.

And then the guy, he thinks about the fuzzing list, my my orchestrator friend. And then he's like, "Oh, we need someone to go after this end point and he makes the whole work." I love that guy. >> Yeah. >> Me and me and Hail Mary mode are like best friends. >> That's so funny. I feel like I haven't read enough of the Chain of Thought of the Hail Mary mode. >> I'm like all about reading Chain of Thought. I That's That's what I I love reading Chain of Thought. >> Cool, dude.

All right. Well, uh, since we're a few minutes over, I will go ahead and call it here. Um, I do want to uh either on the podcast or um at least in the critical thinkers chat kind of debate this whole discussion around hackbots and like the models being blocked off like you know we all lost access to fable which it was while I was on vacation so I didn't even get a chance to use it. I feel like I like had to improve some skills and that was it.

And I really want to use it uh to improve our whole system but >> um I'm going to tell you something. I have one little say on that is that I I can be completely off on this matter, but I think that is there is going to be come a point where it is a better hacker than I am um at everything. You know, that's and and I'm dreading that point because then that means that there's no purpose for me anymore. So like I like when when Mythos and Fable like weren't allowed um I don't know better than me. >> I wasn't like heck yes because I do also want to play with it.

And I also am somebody who likes to play with it. But like maybe the the as I like hacking a lot. I don't want it. It's I don't want it to I don't want it to. Anyway, that's that was my little I wasn't I wasn't upset. I wasn't that upset about it. I because uh cuz um Opus >> me and Opus are still better at a lot of things >> together than Opus is alone. >> Um and maybe that won't happen later and that and then I'll be a sad >> Yeah.

No, I think that'll last a long time though. At the very least, people to orchestrate stuff, to build hack bots, stuff like this. And you know, you've already got your you've already built up six months of that skill set. Uh, which is pretty cool. >> Yeah. All right, dude. Thanks for hopping on here. Have a good day. >> Anytime. >> Peace, dude. >> You know, I love it. >> And that's a wrap on this episode of Critical Thinking.

Thanks so much for watching to the end, y'all. If you want more critical thinking content, [music] uh, or if you want to support the show, head over to ctbb.show/isord. You can hop in the community. There's lots of great highlevel hacking discussion [music] happening there on top of master classes, hackalongs, exclusive content, and a full-time hunters guild. If you're a full-time hunter, [music] it's a great time. Trust me.

All right, I'll see you there.