Image Reconstruction From Human Brain Activity transcript

This image was shown to someone whilst they had their brain scanned and just by looking at their brain data it was predicted that this is the image they were looking at. I mean, look at that. Is that not insanely impressive? Yeah, it's a bit grainy and definitely far from perfect, but this was the first ever picture taken and look how good those have become. So, brain scanning mind reading technology is kind of a thing now and I'm going to tell you about it.

Let's get into it. Light passes through our eyes and hit specialized cells at the back of the eye called photoreceptors, which convert the incoming light into electrical signals that are sent to the brain via the optic nerve. Information is then transmitted to the primary visual cortex located here in the occipital lobe at the back of the brain. Now, the exact way this information is encoded at the back of the eye and transmitted to the brain is still kind of a mystery, but we know that somehow, in some form, the brain has encoded an image.

The encoding is extremely complex because every image we see is mixed with our own feelings and biases. These feelings and biases lie in many different parts of our brain. For example, seeing a teddy bear. Do you have a positive or negative association with a teddy bear? Do you have smells or memories or specific people you associate with them? All this information is somehow encoded in your brain. How exactly? Unclear, but it's represented there in some form and we need a way to decode a brain's representation of an image.

This is basically an attempt at mind reading. Converting a brain's storage of an image to a picture that we can look at, which brings us to this paper demonstrating one exciting solution. Their process of this decoding process starts with an fMRI brain scan, a functional magnetic resonance imaging brain scan. In your brain, blood with oxygen flows to more active areas, so these areas can access more energy. fMRI brain scans can detect this change in blood flow, showing the more and less active parts of the brain whilst you do an activity like staring at an image.

All of this information about your brain can be collected into a brain activity map. What does that even look like? Hold my beer. First, let's imagine this in 2D. Here is a 2D slice of a brain. You place this inside a rectangle, then divide this large rectangle into tiny rectangles where each tiny rectangle is given a value for the activity level at that specific location. Every tiny rectangle has its own unique value for the activity of the brain within that area.

So, now imagine this in 3D where the whole brain is a 3D cube split into tiny 3D cubes where every tiny 3D cube has a value for the brain activity in that area. Each of the tiny 3D cubes is called a voxel. You can then have an entire 3D brain cube image taken every 2-ish seconds for about 8 minutes, creating hundreds of images of this 3D brain structure. This gets so cool because now you can track a single 3D cube across time, telling you how brain activity changed in that tiny cube of the brain during a certain activity.

Tracking all of the cubes tells us how all brain activity changes across time. Okay, does this make sense? The person is lying in the scanner looking at an image on a screen above them. A 3D picture is taken off their whole brain every 2-ish seconds for about 8 minutes whilst they just stare at this image. What you get from the fMRI scan is multiple 3D images of the brain across time, each split into little cubes. You can then look at any little cube and see how brain activity was for that particular tiny cube.

Like, was it high or was it low or was it medium? And how that activity changed across time. Okay, so a lot of information. Mind cleansing brain joke. What do neurons do on their birthday? They cell-e-brate. Okay, we're ready for the next step. So, we have our huge 3D structure of cubes representing brain activity. Remember, there are hundreds of these in total. One 3D whole brain picture taken every two-ish seconds for about eight minutes.

Again, let's simplify this a little to a 2D 3x3 grid where each block represents activity levels of a part of the brain. To help intuition, maybe this represents the front of the brain, this is the central part, and this is the back of your brain. So, the person looks at an image of a lovely tree during the fMRI brain scan, and the scan tells us that their brain activity grid looks something like this. Lots of activity here, some here, nothing here.

We can say that this brain activity pattern represents tree. Now, they look at a teddy bear, and the scan tells us that their brain activity looks something like this. We can say that this brain activity pattern represents a teddy bear. Remember that these brain representation patterns are really much larger in three dimensions and across time, so really four dimensions, but the principle stands. We found a way to connect an image being looked at and brain activity whilst looking at that image.

Now, we get hundreds of people to look at a hundreds of different images across hours and hours of brain scanning time to create a dictionary of images and associated brain activity. Machine learning can then be used to train a computer program to recognize that this is a tree, this is a forest, this is a boat. So, we end up with a computer program that's kind of learned the specific images associated with different brain activity patterns.

You You then give the computer program new brain activity data of someone looking at an image it hasn't seen before. The computer program, in theory, can then use all this information it's been taught to make a guess of what image this brain activity data represents. And some of the results are absolutely mind-blowing. This paper published in 2022 used a novel stable diffusion model on data acquired in 30 to 40 scanner sessions across months where each subject viewed 10,000 images three times.

They produced these results. Aren't these stunning? Anybody can tell that this is a teddy bear, but it's really this clock tower that gets me. The computer program even predicted the little circle clock at the top of the tower. How freaking cool is that? I didn't realize this was actually a thing, and I'm actually doing a PhD using fMRI. There are other studies similar to this with some really cool results, but I think this showcases this technology the best so far.

And the better the scanning and training gets, the more accurate the guesses of the computer programs. There have also been attempts to apply this to videos over the last couple of decades. This paper showed that in concept, this is possible with video, and the results are really impressive, but I don't think the images are quite as detailed. But the quality of the image and video predictions will only get better as this research progresses.

To me, it seems possible that this same concept of scanning and training and testing could be applied to other things. Memories that people have, how someone feels, or even maybe what someone's thinking. It's worth noting there has been some unsuccessful research. Well, no research is unsuccessful, but they didn't manage to achieve it, of research to recreate images people are only thinking of. But the brain activity signal acquired isn't strong enough to make accurate predictions.

But this technology is currently in its infancy and soon it will be all grown up and portable and powerful with enormous and phenomenal capacity for both good and evil. Like imagine the benefits of people with speech and communication difficulties being able to communicate more effectively. That would be an absolute game changer. But there's also weird and crazy ethical issues. Like say someone had a thought to murder someone.

Do you lock them up? We all have thoughts for which we're thankful that nobody else knows. Imagine if big corporate companies like Google and Apple and Facebook and TikTok had the power of mind reading to target their products to sell us more stuff that we don't need. Our data on how we use products is already powerful enough. So imagine if they could buy access to our actual brain data. Anyways, what do you think about all of this?

Do you think this will be a force for good or evil in the world and do you think it will progress as I've predicted? Thanks so much for watching. As a PhD student, I'm trying to improve the quality of my videos, but my budget is limited. So I've actually set up a Patreon. Basically, I want to buy equipment to improve lighting, sound, and set of these videos. If you're interested in supporting me, please go to patreon.com/zacharycortex.

As a thank you, I will include your name at the end of each video. Anyways, thank you so much for watching. Let me know if you have any questions and I'll see you in the next one. I want to buy light equipment to improve the lighting. Light equipment to improve the lighting.