Lecture 9 Fall 2025: Principal Components Analysis transcript

All right. Hi everybody. So, um, today we're going to have a short lecture basically covering the principal components analysis stuff that we didn't get to last week. And then, um, the rest of the lecture will be just me answering any questions you might have leading up to the midterm. So, midterm review. Um but before we get to that I would like to talk today about principal components analysis. So we're going to consider um a matrix data x that we're going to write as n by p.

So we've got n points in p dimensions and the first step with PCA is to center the columns. And so basically what I mean by this is we are going to take the J column and we are going to subtract off its mean. So X colon J minus the mean of all the entries of X colon J. So what this is doing um or I guess just to be really concrete about this x colon j minus 1 over n sum from i = 1 to n x i j like that. Okay. Now what's happening with here geometrically is if my original data so let's say that p is equal to 2.

So here are my x1 coordinates and my x2 coordinates. Imagine that when I plot them they all look something like this. And so by centering all we're doing is we are making it so that they are well centered around the origin. And so we've got the same kind of overall distribution of points. We've just shifted it. Once we've done that centering, then all that's really left with PCA is to compute the SVD. So after centering, so I'm just going to call my new centered matrix X um I compute the SBD U I'm sorry X= U sigma VRpose and then the right singular vectors.

So the basis vectors that are in this V matrix are called the principal components or sometimes people will call them the principal component directions um of x. Okay. So, we aren't really doing anything different than what we've talked about before. We're just using slightly different vocabulary. And in particular, we looked at this example or this distribution of data before. So, if we were to do PCA here, then this would be my first principal component direction V1.

And my second principal component direction would be orthogonal to it like that, V2. Okay. So these are just our right singular vectors but we can call them the principal component directions. And I think it's good to know this vocabulary because it shows up a lot in the literature. But one thing that you will see in many different descriptions of PCA is an alternative way of computing the principal component directions.

So alternatively we could use something called the iigen decomposition. So specifically what people will do is they will start with the matrix the gram matrix xrpose x. So this matrix is now p by p and then they will try to perform an ien analysis on this gram matrix. It's going to result in exactly the same thing. I'm just going to try to connect here the way that we've talked about principal components analysis so far with a very common treatment that you will see in many different textbooks um and papers.

So more generally let me try here we go to talk about iigen analysis and then we will connect it directly to principal components analysis in just a second. So we're going to just consider a generic square let's say P by P um matrix. So when we do an igen decomposition we are looking for IGEN values and IGEN vectors. So we say that lambda a scalar is an igen value of a. If there is a vector V which is also a P-dimensional vector where this vector is not equal to zero so that we can write the product of a times this vector and what we get out is a scaled version of that vector and the scaling factor vector is lambda and in this case we call V an IGEN vector.

So we've got IGEN value and IGEN vector pairs here. If we have P igen vectors and I'm going to just put them all together in one big matrix V. So I'm going to have V1 V2 all the way to VP and Pigg values that correspond to them. And I'm going to put these all in a matrix capital lambda on the diagonal. So I've got lambda 1, lambda 2 all the way to lambda p. Then we can take this equation here that will hold for each of the igen vectors and we can write it simultaneously for all of the igen vectors and igen values and say that a * this matrix v is equal to lambda * uh v.

I think I got that wrong. A times yeah B lambda. Okay. So now we've got a collection of igen vector and igen value pairs and we can put them all together in one matrix equation like this. And in the special case when a is orthog uh symmetric. So when a is symmetric and remember that means that a is equal to a transpose then v is orthogonal. Um you know what I'm sorry I'm just running out of space here. So, I'm going to move over and make sure there's plenty of room.

Okay. So if A is symmetric then V is orthogonal. And remember that means that V transpose V is equal to V. VRpose is equal to the identity matrix or in other words VRpose is equal to V inverse. And so we can take that expression A * V is equal to V * lambda and we can multiply both sides of it on the right by VRpose. And so we get a vranspose is equal to v lambda vranspose or in other words a our original matrix can be written as the product v lambda vranspose.

Okay. So now we can go back to what we were talking about before with principal components analysis. What I told you is that with PCA we would start with Xrpose X and we would do an igen decomposition. So in the context of what I just described, this XRpose X is our matrix A and it is symmetric. And so what we're going to do with an igen decomposition is have x transpose x can be written as a vector of igen vectors a diag I'm sorry a matrix of igen vectors a diagonal matrix of igen values and a matrix of igen vectors transposed.

Okay. So, how is this related to what we were talking about before when we said we would compute um principal components using the SVD? Um well, okay. So, first of all, when people talk about computing PCA using the igen decomposition, what they will do is they will compute XRPOSEOSEX. Then they will do an igen decomposition a a routine that's in numpy for instance to get these um matrices v and lambda and then they will say that the columns of v are the principal components.

Okay. So it perhaps is not obvious that these two things are equivalent. So what we're going to do is we are going to take this matrix xrpose x and rewrite it using our svd components. So xrpose x is u sigma vranspose transpose time u sigma vranspose which is v sigma transpose urpose u sigma vranspose how can I simplify this h yeah >> transpose Exactly right. Uranspose U is just the identity matrix. And so what I'm going to have here is V sigma transpose sigma time VRpose.

And so now this sigma transpose sigma this is exactly the same as what um as the what we've been calling lambda. And so these things are equivalent. We're getting the same principal component directions. And the igen values are the um squared singular values. Are there any questions about this? Yeah. Um, are you looking here? Here. Okay. So, you said when I have x, this >> Oh, this is supposed to be a lambda. Yeah. Any other questions?

Yeah. >> Could you explain why it's a square values? >> Yes. Um, let's move over to here. So when I have my sigma matrix, let's consider the case where n is greater than p. The number of samples I have is bigger than my dimension. So this is rectangular. So I'm going to have sigma 1, sigma 2, all the way to sigma p. And then everything else in this matrix is um zero valued. And so now if I were to look at sigma transpose sigma, then I'm going to have something that's p by p.

So it's going to be square. And the diagonal entries are sigma 1 squared, sigma 2^ 2 all the way to sigma p^ squar. And so this is a diagonal matrix. The entries are the singular values squared. And I'm just observing over here that when I've written xrpose x in terms of the svd components, I get this product that has exactly the same form as the iigen decomposition form. >> Yeah. Yes. >> So the like the right singular vectors are just the vectors.

The right singular vectors of X are the igen vectors of XRpose X. Now I want to be clear here that um you know this form that we're talking about depends on that matrix A or in this case XRpose X being symmetric. Of course when you have Xrpose X it is symmetric. Um, but I'm just saying what I'm trying to say is that if I had a generic matrix A, there's some additional subtleties to IGEN analysis that we're not going through here.

We're really focused here on the case where the matrix we're computing the IGEN decomposition of is square and symmetric and real valued. Okay. So hopefully then that'll help your understanding. If you see this treated in a textbook or a different tutorial or something and you're thinking, "Oh, we only learned about this in terms of the SVD." This is an alternative framework. One of the reasons that people like this framework is because when they look at this matrix XRpose X, you can think about it from a statistical perspective as a covariance matrix.

So we talked before about how when we think about principal components analysis or the SVD that first basis vector V1 is capturing the direction in which there's the most variation among our samples. And so I can think about XRpose X as giving measures of the variation among my samples in different directions. And so from that statistical perspective, many people find it much more natural to think about factoring this xrpose x matrix as opposed to starting with the SVD.

But mathematically they're equivalent. Any other questions? All right. So now what I want to talk about is some of the ways that people use principal components analysis in the context of machine learning. Okay. So most frequently principal components analysis is used for dimensionality reduction. And so what do I mean by that? The idea here is we want to replace each sample x i which is in p dimensions with a new vector and I'm going to call it zi which is in k dimensions. for K being much smaller than P.

So for instance, maybe I could think about having a collection of photographs of different people, each of which has P different pixels in it and I'm only try and I only Yeah. And so there's going to be a lot of commonality among them. And so what I could do is I could say, well, are there some features within that collection of photographs where I need many fewer features than the collection of all pixel values in every image, right?

So I might want to reduce the dimensionality that way. Or you could imagine a setting where perhaps later I want to use this data to do something like le squares but some of my features are linearly dependent or close to linearly dependent and I want to remove that dependency. So I can try to reduce the dimensionality so that the features that I'm left with are no longer linearly dependent on one another. often we will want to preserve some important properties.

Um, so for instance, um, Huh? I might want to say, well, if I were to look at any pair of samples, let's say X I and XJ, and look at how far apart they are from one another, perhaps what I want is that after I reduce the dimension, the corresponding points ZI and ZJ are the same or approximately the same distance apart. So even though the actual vectors are different, they're still kind of keeping the same. If points are similar in my original space, then they're similar in my new space.

And if they're really far apart and dissimilar in my original space, then they're far apart and dissimilar in my new space. So I might want to do something like that for instance. Okay. And so PCA is going to be a tool that can allow us do that to do this dimensionality reduction. So with dimensionality reduction, we can do all kinds of things. We can reduce data storage or our memory requirements especially when P is enormous and K is small that can be very impactful.

We can also better visualize data. And so often you will see in papers where people will have highdimensional data but they're making a scatter plot in two or three dimensions. How are they doing that? Well, first they are somehow reducing the dimensionality of the data. Um people don't always use PCA for that. or other visualization techniques but they are somehow reducing the dimensionality of the data and PCA is one way that you can do that dimensionality reduction and another one is you can um reduce um correlations and by this I mean having features that are nearly linearly dependent um among features Okay.

So let's talk about now how we can accomplish this using PCA. Okay, so we are going to have our matrix X have our samples in rows. So I've got X1 transpose all the way to XN transpose just like we've been doing all quarter. So this matrix now is N by P. And our goal is to take this and transform it into a new matrix Z which is going to have rows Z1 transpose all the way to ZN transpose. So we've got the same number of rows, but now each of those rows has only got K entries where K is much smaller ideally than P.

And so what we are going to do is compute the truncated SVD of X. So if you recall from last time we wrote this as x k which was the product u k sigma k vk transpose and in this sigma k was a k by k diagonal matrix. So we're only keeping the top k singular values. U K is going to be an N by K matrix. So I now have um yeah K different elements in each uh or K different columns in U and VK is going to be P by K and then we transpose that.

So the product V branspose is K by P. Okay. Okay, so X K still has the same size as X. It's still N by P. So we haven't really ch reduced dimension yet. But this matrix X K now has all of its samples, all of these different rows are lying in a kdimensional subspace. So let me write this over here. um x k is the same size as x and that is n by p but each row of x k is in a kdimensional subspace. So to illustrate this, right, imagine that we have a whole bunch of points that are in 2D like this.

And then we compute X K. X K is going to give us a whole bunch of points. Now, same number of points, but they all are going to lie exactly on this subspace. Maybe that's hard to see. I'm just going to draw a solid line here. And so, X K is going to have all of the original points projected onto this line. So, they're still being represented in a 2D space, right? We still have P in this example is equal to two. but they all lie along a one-dimensional subspace.

So here K is equal to one. So now that we have this, now that we've done our truncated SVD, we can say, well, if my points are in this kdimensional subspace, then I don't need to use P numbers to tell me where. I can represent the location in the Kdimensional subspace. just using um k dimensions. So in this little toy example we made what we're doing. There we go. In this toy example, if I wanted to tell you where this point was located in the one-dimensional subspace, I could either give you the two-dimensional coordinates of that point or I could just tell you where this point is on this line relative to the origin.

Right? I only really need one number to tell you where that point is if you already know it's in the subspace. So I can reduce the dimension, reduce the number of numbers I have to tell you to communicate where that point is. And that's the basic idea here. So in particular what I am proposing is that we can let Zi be equal to um sigma 1 * I always get this okay UI 1 sigma 2 UI 2 all the way to sigma K UI K. So all that's happen and this is a K- dimensional vector.

So all that's happening here is I'm saying after I have done my truncated SVD, my points lie in a K- dimensional subspace. And so I need to and that subspace is spanned by the K columns of VK. And so what I need to do is I need to say well what are the basis coefficients or weights on the basis vector that I use for sample X I and those weights are just coming directly from the truncated SVD. And so this full putting all of these pieces together the full matrix Z can be computed as X times this VK matrix or equivalently and more like what I have written above U K * sigma K.

And if I do this then what I get is a matrix that has n different rows one for each of the different n samples that I care about times k different columns because now I'm representing things just in terms of its location in this kd dimensional subspace. Let's just look very quickly at another um geometric example. It's a little harder to draw in 3D. the written notes online will have a 3D example, but let's just imagine that all of my points are not exactly in the 2D plane, but they're close.

And so for each one of these different points, I can kind of project it onto a point in the 2D plane. So the white point here would be like x i. And this point here is the projection of x i onto this 2D subspace that I'm just going to call s. All right. So both of these points are in R3. But now once I've projected all of the points onto the 2D subspace, then all I need to tell you is where they are located within that 2D subspace.

I just need to give you two coordinates. I don't have to tell you the vertical coordinate because I've already told you they're all in this 2D subspace. And so how would I do that? Well, my singular value decomposition or my principal components are telling me that this data is varying a lot in sort of the uh in in the horizontal plane and barely varying at all vertically. So, the top two principal components are going to be in these two directions or maybe a rotation of that, but they're not going to be vertical in this example.

And so when we do this um uh dimensionality reduction via PCA or via the truncated SVD, what we're doing is we're saying, well, we're only going to represent each of our points in terms of where they are in this subspace spanned by my first couple of principal component vectors, and I'm going to ignore any information about where they are in the other basis directions. Are there any questions about this? Yeah. >> So like why why are we using like this like Z instead of just the truncated aren't they representing the same information? >> So with the truncated SVD when we computed this we are getting a matrix that's N by P still.

So we haven't act. So each of the rows of x k here are in a kd dimensional subspace but I'm still representing them with p coordinates. And so I'm so in our little 3D example over there, it's like each one of my here maybe I'll just write this out, right? like x i could be for instance I don't know 10 -6 and then2 and then when I do the truncated um svd then I'm going to get an x i put a little superscript k here to correspond to this is the kth throw in the truncated svd what I'm getting is its projection onto this 2D plane so that would be like 10, -6, 0.

And in fact, all of the different points in that truncated SVD in this example would have that third coordinate be equal to zero because I've projected them all to that horizontal plane. And so what I'm saying is, well, instead of keeping this full vector, I'm just going to keep the parts where I've got variation. So this is going to be my zi. So this is actually lower dimensional in terms of like the number of entries in the vector than the rows of my truncated SVD matrix.

Any other questions? >> Yeah. >> Did you write truncated? >> Yes, that's supposed to be an abbreviation for truncated. a very very poor abbreviation. >> Yeah. >> Uh like in this example you use the coordinates. >> Yeah. So does it have to be the coordinates or it can also be >> Yeah. In general it would not be the coordinates. In general, the directions that you would find would be some yeah, arbitrary rotation. Um, I was just using this example where they correspond to the coordinates to make the idea hopefully clear like why we can get away with reducing the dimension after we've projected onto the subspace.

It's just less obvious if you have an arbitrary rotation of the coordinates. the ideas, all of the formulas and everything that I've presented would have you could have any arbitrary basis for these different subspaces. They're almost never actually coordinate aligned in real life. >> Yeah. Can you explain again how you get from XVK to to product between UK and C? Okay. Like I understand XVK because we said it's the span of um the rows of B and and every row of X gives the weight.

So but I don't understand how we go from that to >> Sure. Yeah. Let's just walk through that. So x is equal to u sigma vranspose. And um let's just remind ourselves that when we do the truncated SVD we are taking our V matrix which is P by P. So V is in R P by P and then we are keeping the first K columns of that. So this here is V K and it is P by K. But obviously it's related to V, right? And so now if I were to look at X * V K then I have U sigma VRpose time VK.

Okay. So maybe to make this clearer I'm going to call all of these other columns here V tilda. So now V is equal to the concatenation of VK and V tilda. And so I'm going to plug that in over here. I'm going to write U sigma. And now I'm going to rewrite V as the VK component concatenated with this V tilda component. Transpose VK. Now what I can do is I can multiply this VK on the right by each of these two components separately.

So I get U sigma and then I'm going to get VK transpose VK and V tilda transpose VK. Now by the way we constructed this remember that V is an orthogonal matrix. So V transpose V was equal to VVRpose was equal to the identity. And when we do the truncation we have that VK transpose VK is equal to um the identity. And what this orthogonality means is that all of these columns in this V matrix are perpendicular to one another.

And so if I were to look at VK transpose V tilda, that has to equal zero because all of these VK columns are orthogonal to all the V tilda columns. And so what I have over here is now oh sorry uh yeah okay u * sigma times now now I've got an identity and this matrix of zeros. So far so good. Okay. So now this is equivalent to UK and sigma K. And to see that I think it's easiest to kind of draw similar sorts of of pictures.

So I've got remember that U was similarly like a square matrix. It was n byn but uk k corresponded to just the first k columns of that. And similarly sigma was an n byp matrix and sigma k was just the first k by k block of that. So over here when I have u * sigma time this concatenation of an identity matrix that is k by k and a zero matrix what's happening is that that identity matrix times sigma k is multiplying all of these singular value or well I'm sorry the identity matrix that's k by k that part is preserving ing all of the singular values that are in sigma k and everything else is getting multiplied by zero and those zeros are then getting multiplied by these extra columns in u that are not in uk.

So the only things in this product that are not being multiplied by zeros are going to be uk sigma k. Yeah, great question. Any other questions? Okay. So, in um the lecture notes, we have one numerical example. Um it's just really showing a specific matrix X and then computing U2 sigma 2 for that matrix X. I'm happy to walk through it, but if people have burning questions about the midterm, I'm happy to focus our time on that.

What would people prefer? Yeah, >> sure. >> So, why is >> Oh, did I do that wrong? >> No. Um, yeah. VK is P by K but then V uh the product is going to be K by K. Yeah. >> Yeah. >> Could you explain why like we have transpose? Um so this product is equal to zero because when we first talked about the SVD we said that this whole matrix V is an orthogonal matrix which means that every column of V is orthogonal to every other column of V.

So every column in the VK matrix has to be orthogonal to every column in the V tilda matrix. >> Yeah. Do you know why the interior is also orthogonal? >> I'm sorry. What's that? >> In a b lambda like the upper >> Oh, okay. Sorry. Say the question again. >> Why is B and like when we're talking about the the demposition, >> why is that also orthogonal? Oh, so we haven't proved that they are orthogonal. I'm just making that claim.

Um, there's some additional subtleties with identity decompositions that I'm not going to go into. Yeah. >> Yeah. When people talk about orthogonal vectors, they just mean perpendicular like the inner product is zero and nothing about the norm. But when people talk about orthogonal matrices, they mean that the vector the columns are all orthonormal. So when you talk about an orthogonal matrix, that normality is implied.

And I know it's not super intuitive, but here I was saying that the matrix V is orthogonal meaning specifically this that vranspose V is VVRpose is the identity. That is what orthogonal matrix means. It's an easy thing to get confused on. That's fooled me before too. Um but yeah, this this normality is implied when people say orthogonal matrix >> so we are like just regular SVD. So we are making sure that B is orthogonal matrix.

That's right of the columns are >> exactly whenever we do the SVD we are always finding a matrix V and similarly U that is an orthogonal matrix by definition and so all of the columns are unit norm and orthogonal to one another. Yeah. >> Yes. >> Just to clarify, all the columns of V are >> all of the columns of V are orthogonal to one another and have unit norm. >> And we're doing singularation, why we like use like only the columns of B as the as the singular vectors instead of the columns of is that we want to fit to x instead of fit to the feature. >> So the way we're doing this is we are saying that um x is a matrix that's n by p where n is the number of samples or say examples and p is the number of features in each one.

And so I think about this as being a collection of end points in a pd- dimensional space. And so then what I'm trying to represent or what I'm am representing with V is a basis for that pdimensional space. Now I guess U is here a basis for an N- dimensional space. So I could kind of turn things on their head. Um, but I think this is the interpretation that's perhaps most intuitive within the machine learning context that we're considering here.

Um, but this is a good point. Um, in general, so because sometimes people will write their data matrices differently. They'll have it be P byN for instance. So if someone were to give you a matrix that's p byn and you want to find the principal components and you do xrpose x you'd get the wrong matrix. So in general the thing to think about is the geometric perspective where what we want is to have a basis for the pdimensional space of features.

And so if you're going to use this igen decomposition approach for instance, then what you want to do is to make sure that whatever matrix you're computing your igen decomposition of is p by p because you want a basis for the pd dimensional space. If it's n byn, if you're getting n- dimensional principal component vectors, then that suggests you're doing something not quite right. Um, so I'm trying in this class to be consistent and to always write X as being N by P.

Um, just so that it's clear from lecture to lecture what I'm doing. But of course in real life, people are just going to give you like a big data file. And it's your job to figure out how to actually use these tools, which might mean just like massaging everything until it's N byP and then you can just use things straightforwardly or thinking a little bit more deeply about what the methods mean. Yeah. Um, I noticed