Exploiting Containerization | Bug Bounty Hacktivity Explained transcript

Containers are a pretty complicated piece of software engineering technology and nothing really shows that best than the people who try to exploit them. And in today's video, I'm going to focus on exploits found [music] in containers and apps that utilize containers for their development process. And if you're new here, this is Activity Explained, the series where I look at books found by normally other bug bounty hunters so that we can analyze them and learn from them.

And the goal is not to just admire the vulnerability and understand, you know, understand it, but to actually dig [music] dig deep and figure out its root cause like why the developers made this mistake. What assumptions did they make that caused it? And those are things that are very important to understanding vulnerabilities and actually applying them [music] to our own bug bounty journey as well. And our first report is in [music] an application called SEML.

I think it's SEML. It was semi. I'm not sure. Either way, this [music] is an application or was an application because it was acquired by GitHub where basically when you had your source code, you [music] would uh have it there and when you sent it to SEML or SEMI, they would build it into a Docker container and then run vulnerability analysis on it inside like this containerized environment. [music] So, it was kind of like a unit test if you know what that is, if you're a programmer, just on like the whole application from a security perspective.

And the way you told Seml how to build these containers is with this lgtm.yaml file which is probably like a very primitive docker file or something that looks like it. It looks a bit like it as well. And this basically told SEML how the application should be run in a way that's you know in like the actual like um environment for the application because different applications are going to require different setups, different databases, different you know backends or whatever.

So having a file like this is effectively essential to correct vulnerability analysis. The other thing is that they had like a web application. So they had a web application where you could actually view the docker container and you could view specifically the source code that you actually uploaded. So let's say some developer uploads a source code to this seml application and there's some vulnerability testing going on then some manager could log on and be like oh this is this got uploaded and then he can read all of the files that were uploaded you know like a verification check or whatever including this lgtm.yaml yaml file basically that was included and basically you could have this in two forms so you could have either this lgtml file or you could have lgtm.yaml the ammo file.

So in Linux, if you don't know the dot before a file means that the file itself is hidden. So if you open it in like a command in like a folder or you do ls on the on the directory, it won't show it to you unless you explicitly specify that you want to see hidden files, which in Linux is the ls- a command. That'll show you all the files. That's important to realize. So these these configuration files often have books.

These configuration files are often buggy just because developers kind of, you know, it's it's hard to implement this. You you can't really blame them that much for it. And there's a lot of different things that can go wrong here. And in this case, the thing that went wrong is with sim links. The developers kind of forgot about sim links. What is a sim link? In Linux, you can have two types of files. just a normal file with its contents or you can have a sim link which points to the contents of another file.

And basically most commands if you try and read a sim link it will actually read the contents of the file it is pointing to just by nature of the operating system redirecting you to it. And in general, you have to specify specific flags on commands for it to explicitly either not follow sim links or like not read the sim link or output the actual content of the sim link which is usually just the actual absolute path of where the file is.

And developers forget about this mainly because you know you have to do it requires configuration to not do which are the best vulnerabilities are what the developers have to take steps to prevent them. We don't want to vulnerabilities where the developers have to take steps to cause them. The ideal vulnerability is where a developer has to take steps to prevent the vulnerability because chances are they're going to forget because they're like, you know, overworked and burnt out and stuff like that.

So, we want to abuse that as much as possible. And basically, this hacker figured out that when you created one of these configuration files, specifically when you had two of them at the same time, so there was two options, the LGTM.YAML yammo and the lgtm.yammo. And if you included both of them, the program would get a bit confused and use the first one, but still kind of still kind of process the second one in in some way.

And when I when I mentioned the file viewer, what the program actually did was when you uploaded the source code to it, it automatically copied that source code into another directory or folder or whatever so that it could display the files to you. But what happens if you copy a sim link? If you copy a sim link by default in Linux through the command to copy a whole directory, what will happen is the sim link will be resolved to the file that it's pointing to.

And this is a bit of a weird design on Linux. I think that this is probably a little bit dodgy, but if you uploaded a sim link as this.lgtm.yammo YAML file, what would happen is when you went when the program went to copy the directory, that sim link would get resolved. And if you had that sim link pointing to a file on the host machine that you were not supposed to read, like in this case, the etc/ file, when the program went to copy it over, it would literally include the contents of the file in the sim link, which means that this is a local file read vulnerability.

And that was obviously marked as critical because you could read any files which is pretty cool if I mean this is this is really great if you can just do this that just seems like honestly if you can ever get like this where you can just say hey I have your etc/pass file I think that's like mad or if you can get something like this that just like you know give me that critical vulnerability you know [laughter] but yeah that's pretty cool and you set up the project uh to make a sim link on Linux I think you just have to do like ln and then the file and then the file you wanted to link to.

So, it's pretty handy. Now, one thing that I want to say is that, you know, I was doing some research for this video was that this is not the first time that this has happened to the same company from the same hacker. I managed to find this second report that was literally submitted a week before the other one. So, this is the one that we just covered. This one was submitted a week before the other one and it's literally the exact same vulnerability.

Basically the exact same vulnerability, the same attack. It's basically instead of this um using this configuration filelgtm.yammo instead it uses a log file that was automatically generated by the server like here this uh out option.out.napshot.log.build.log log which was accessible to the application. So the docker inside a docker container and what it it basically did is during the building of the inside the current configuration file where you tell the docker container how to build your actual application you just say to switch that log file to a sim link instead and that sim link is going to point to the etc/ file and this log file also gets copied over by the application and when it does that it also resolves the sim link because you know by default it usually will resolve the sim link and I think that's that's um I think that's very funny because it's literally like the same attack and it happened to the same company within a week.

So this is really showcasing you that when there's like one kind of mistake [snorts] a developer made, it's also likely going to be, you know, in the other areas. And really the root cause of the vulnerability is developers assumed that the files that they were that were on their machine were going to be safe that copying them over there was not going to be any risk of user input being in there because they're already on their machine and they've been like presumably like sanitized or whatever.

But that assumption was wrong because you can include sim links and files and the functions on Linux. So the commands on Linux automatically resolve them. So that's how this mistake happened. And I think it also does teach you that whenever there's one developer coding something, it's probably the same guy that coded this, by the way. Whenever there's a developer coding something, if he made one assumption in one place, he probably made that assumption other places, too.

So it's worth checking it. And you can literally get two backtoback criticals like this fell did. Like that's a very cool vulnerability rewarded $2,000. This guy test and null which is very interesting and it's a privilege escalation which is probably not correct but yeah two very nice vulnerabilities shows you how you can use the functionality of containers against the application itself. Now, this next report isn't like really a bul bounty report like in the traditional sense.

This wasn't actually reported to any programs, but it is a really nice crash course on how to actually do privilege escalation on containers, how to basically escape containers and infect the host. And in this case, the target was Google Cloud Shell. Google Cloud Shell don't know it's like this application where you basically just have an online web shell and you can use it to code your website. You have like a Visual Studio Code thing in there.

You can access Google APIs which is I think the main reason people use this is you can code like small Google API apps like Drive apps or whatever and it's a bit easier to do than doing it normally because your API keys are already included and you can just make a request directly. That's kind of nice. And basically in this application when you got onto it looks something like this. And you start off with the root privileges on this on this uh container.

So you're already the root and you can do pseudo su on the container. Now in these kind of applications you're kind of expecting two things, right? One, you're going to be in a docker container, right? That's pretty obvious that they're not just going to let you get a shell on the host, right? So this is most likely a container. The second thing you can probably expect is that you're in a Kubernetes node. So Kubernetes is different from Docker.

Kubernetes is a way of managing multiple Docker containers using something called nodes. So Kubernetes is a bit complicated, but essentially the server that you're on is the node and then the actual container plus all of its like services like maybe databases or whatever. It's called a pod in Kubernetes. So when you're on a node, it's going to spawn multiple pods inside of it usually. And you know, when you're inside of a container like this, you're going to presumably be in a Kubernetes pod.

And the way you can figure that out is usually you can check if it's a service account or whatever inside of it. But you can also see if it's pointed on a path like this. So if you run like proc one/cgroup or proc self/cgroup I recommend running it would show you basically that you're inside of a cubernetes instance is k is just a classic cubernetes path and this is a lot harder to escape than docker is is to escape mostly because kubernetes containers tend to be on different servers.

So if you want ultra isolation, you have one host running one pod on it and then they're, you know, separated from each other. And the node doesn't have to be like a standalone like brick like server or whatever. It can also be a virtual machine. So the distinction between a container and a virtual machine is that they don't is that in a container the container shares the same kernel as the host and is just like a directory mounted on the actual host itself. while a virtual machine has a separate kernel and it's like a completely separate file system and all that.

So it's much harder to escape a virtual machine than a container. So Kabir is pretty secure in that case and so even if you manage to escape the container, you would have to look to escape a virtual machine after that which yeah, good luck with that. That's very difficult. But anyway, you're inside the container first. So let's look at how we can escape the containers. The first thing that you can do here this this uh uh attacker the hunter did here here is that they checked what kind of privileges they have here.

So in this case they used prox self/ status and check the cap effect. When you're creating a docker container as I kind of showed you before in the last video on docker containers you can assign different privileges and some of them are less powerful than others. For example, you have like um like cap killers. You can like kill different processes and stuff like that. And some of them are very powerful like cap system admin.

You can see here is a very very high permission permission um uh capability that basically allows you to do a lot of things that you shouldn't be able to do. And this is going to allow us to escape the container. When you see this, you can pretty much always escape the container because you have poss you have like permissions to do a lot of things. And in this case, there's a lot of ways to escape now. You have a lot of possibilities and you can basically start from the easiest one and work your way down.

And just the easiest way to escape first is to actually try and load your own kernel. This caps system module capability allows a Docker container to load a kernel onto the host. And this guy like right up on it here. And basically because Docker containers and the host share the same kernel, if you modify the kernel and Docker container, it's also going to change the kernel on the host. And the kernel basically will allow you to get RC, right?

If you obviously if you can upload a malicious kernel, you're going to be able to get RC on the host because that's what like you know runs the system and stuff. So once you're able to do that, you can run your own kernel and you should be able to get RC except there might be a little bit of an issue because in this case there was a kernel signature verification. Now, I didn't know what this was when I was reading this, but I looked into it a little bit and it seems like it's something, as far as I know, developed by Google and it's inside their Kubernetes engine.

So, it makes sense that they're using this technology. And what this does is that when you have a Kubernetes container, the actual kernels that you can load onto it are locked to a certain amount of modules that are cryptographically signed by the Kubernetes cluster at the start. So you can't load any modules that aren't verified by the Kubernetes cluster. And I think the way this works is that the actual engine itself has a private key and it will sign the actual entire contents of the of the what's it called the the kernel into a hash.

And if that hash and that hash is stored somewhere on the server and when you try and load a kernel it will match that hash against the signature of your kernel. And if it doesn't match then it's not going to let it through. I I I think that's how it works, right? Not not entirely sure, but either way, this is something that Google developed, so it makes sense that their systems are using it, right? And in that case, we won't be able to load up any modules.

So, this attack vector is basically dead instantly. Okay. What's another attack vector? The other attack vector is Linux Croups. Specifically, this version one. What is a croup? A croup in Linux is a way of taking multiple threads or pro sorry processes multiple processes in Linux and basically limiting the amount of resources they can consume and this is really what makes containers possible. So if you have a container and you want to limit how much CPU it can consume or how much GPU it can consume then you set up a croup for it and then you put limits on how much you can consume.

Here's a little diagram of how the this is the version one of how it looks inside the actual file system. So you have these resource types here memory disco and your CPU and then in this case you define your groups and then you have processes for each of those groups. And basically this allows you to limit the resources here. So you have groups and then actual resources themselves. This was changed in version two where I don't think they have this anymore just they have the actual group sitting on top and then for each process you actually define a CPU cap and this is what docker uses to limit the consumption of containers and it's also what Kubernetes uses.

Now the attacker here is using a payload here and the idea is that we're trying to target this notify on release binary. So in cgroups version one you had this binary where whenever one of the croups was like released so it was no longer used anymore it would trigger some kind of binary on the host to run some kind of command to run on the host and because we have the cap system admin permissions. Remember this is not possible if you don't have them.

We can actually override the location of that binary with a command on our own on our own um container and then the host will run it. So that's a way to get RC on the host and this is just the command for that just running this and the important thing is that we have to have the actual absolute path for the host on the container. So it's using something called overlay file system to basically create a docker container and we need to find where in the docker container we're actually stored which you can use using I think this thing.

So you can use something like this. Uh where's host path? Yeah, set and per directory whatever. You can use this thing and you can get the host path. But there's a there's an issue. Croups one is deprecated like years ago. So in Kubernetes, the last time this was used on an modern configuration is 2022. So that's about four years ago. So really the the chance of you finding this again is very slim nowadays. in Docker.

This was deprecated in 2020, so six years ago. And basically all Linux distributions run with Croups 2 by default now. And for good reason. They're a lot safer. They make more sense really. They're also like better to manage. So Croups 2 really dominates now. And these vulnerabilities, a lot of these are basically dead nowadays. And in this case, the server was obviously using Croups one. So this way of escaping has unfortunately been patched.

But there is a third way of attacking and that third way is using this thing called hot plug. Now what is hot plug? Well, you have a computer, right? You have your Ubuntu computer running right now and you have a little USB stick. I don't have one next to people. Imagine I have a USB stick, right? And I put it into the computer. You'll see a little notification box come up. Maybe something like USB, the name of the USB stick was plugged in or something like that.

Well, how does that work? How does the contain how does the command run to show that little pop-up box when I actually plug it in? What is something called hot plug? And what this does is basically whenever a new like network device is added, in this case the USB stick counts like a network device. Whenever that happens then this this uh will trigger whatever command we want it to. It will trigger the command that's it's currently pointing to and the important thing is that we can overwrite this ourselves.

So this is just a file somewhere and it's pointing to I think this might actually be one of those sim links where it's pointing to another command and because we have cap system admin permissions and we also have cap system net permissions which basically means that we can actually add both network devices and modify this hop log. We can use this gadget on the host to create a new network device, change the hub look to a command that we have on our own container and then remove the device which is going to trigger this thing.

So basically what the hacker did here is they set the net listener on the thing. So this they were just trying to get a reverse shell. They created a payload for a reverse shell and then basically using the same technique to get the absolute path for the container. They found it here and then simply modified the hop plug to call their own binary that they had on the host which in this case was the reverse shell. And when they added the network device, you can see that it triggered the reverse shell on the host.

And I'm sure there are many more gadgets like this than just these. I'm sure there's more ways to escape containers and really when you create a privileged container you have to you know be be sure that it's running in an environment that's actually you know the container is going to be able to escape because you can just mount the whole file system I'm pretty sure if you want to as well so there's loads of ways to escape these containers but it's also pretty cool though uh showing a little bit away here so you know checking the um the privilege how to do that and then how to actually build a payload.

So I think this this um report is really good for just looking at how to actually you know how to actually practically exploit inside of a docker container and the rest of this was him just trying to privilege escalate from there. So we got root access on the host and he was then trying to break out of the Kubernetes container using API keys and stuff but he wasn't really successful and I'm pretty sure if you want you can actually try this out yourself.

So if you go on Google Cloud Shell, you can do all of these like exploits yourself. So maybe if you find a vulnerability, you can report it to uh Google Cloud VDP. That would be nice bounty if you found it because I actually checked in and the the [music] cap or was it the cap effect was still set to privileged. I'm pretty sure you can still escape if you wanted to. So yeah, you can go try that out yourself. But I actually also recommend actually just reading this report because it was very nice and he talks about some very cool ways to escalate inside of Kubernetes afterwards, which is going to [music] be the topic of a video I'm going to make soon.

And yeah, I hope you enjoyed this video on some vulnerabilities that happened inside containerized environments. I think it's a super good way of generating high impact vulnerabilities in a location that not a lot of people actually look at a vulnerability in one of these containers can easily be scaled to high or critical because a lot of the times it's going to be things like serverside request forgery local file read or you know obviously RC is the big one.

So [music] these are very high impact vulnerabilities and they're very demonstrable impacts. there's no issue getting that high or critical [music] bounty for them. And I think a lot of people do kind of sleep on them as well. Did enjoy the video? Then be sure to leave a like and subscribe to the channel for more book bounty content to be recommended to your algorithm so you can grow and get that edge over the competition.

And anyways, that's it for me. Thanks for watching and happy hunting.