Lecture 3 Fall 2025: Least Squares transcript

Okay, welcome back everybody. Um, so today we are going to really dive into least squares. So what we've been talking about so far is what happens when we try to predict labels associated with um feature vectors uh using weighted sums. And today we're really going to start talking about how we can learn those weights from training samples. So recall that we are given n training samples [Music] which we write as x i comma y i for i = 1 to n.

So x i is a vector of features and y i is a label. And so um each x i we're going to write as being in r p which means this is a vector of real numbers with p elements. And we last time talked about establishing uh some notation and shorthand that we're going to use throughout. So we are going to have a label vector y which is simply going to contain y1 y 2 all the way to yn. So all of our training labels in one vector.

So this is a vector in Rn and we are going to have our feature matrix capital X which is going to have all of the training features associated or all the samp features associated with all of our training samples. So we would have the first training vector X1. We transpose it. So now it's a row instead of a column. And that becomes the first row of this matrix X. And then we have X2 transpose all the way down to XN transpose.

And this matrix of features is going to be in this case real numbers. We're going to have N rows and P columns for the P different features. And so then what we are going to do is make some assumptions here. And we're going to dive into these assumptions and what they mean and why they are important throughout the course of this lecture. But in particular, we are going to assume that the number of training samples n that we have is at least as big as p.

And that, and this is what we're going to talk in more detail about throughout this class, the rank of this feature matrix X is equal to P. Or in other words, X has P linearly independent columns. [Music] Okay, so that's some jargon that we're going to cover and introduce today. Linear independence. And we touched briefly on rank last time, but we're going to talk on it or talk about it in more detail today. Um, but this is going to be essential to what we want to do next.

But intuitively, if we think back to one of the first examples we talked about where we were going to try to classify, for instance, whether an image of a face was smiling or not, we talked about extracting features from that face, such as the distance from the eye corner to the mouth corner. So, you could imagine that we do this on the left side and the right side of everybody's face. And imagine that for whatever reason everybody's perfectly symmetrical.

So we've got two features that are identical for all of our training samples. Then this would present a challenge for finding good weight vectors for making good predictions that we would have to deal with um separately. And so this is where this notion of linear independence comes into play. And again, we'll talk about this in more detail as we go along today. I'm just going to make sure the recording is following me.

Yes. Okay. So, what we want, our goal here is to find a weight vector [Music] W and this is going to be a set of P different weights. So we say that this weight vector is in RP and we are going to choose this weight vector so that if we were to predict a label for sample I by computing the inner product of the feature vector X I with this weight vector and we've written this a couple of different ways including Xiranspose W that that's going to be a reasonably good fit to the training label that we actually observed.

So we want to choose a weight vector that's going to give us good predictions of the labels on the training samples that we have. And we can equivalently write this using our shorthand. So we can say our vector of predicted labels can be written as this feature matrix X times this weight vector W. And we hope that this will be close to our label vector Y. So this is what we would like to do. We want to figure out how to find this weight vector so that this is is true.

And recall that the interpretation of this is that yihat is going to be a weighted sum of the features for the I sample. So weight 1 * x i1 plus weight 2 * x i2 all the way to weight p * x i p. Okay, so this is just a little recap of what we were doing previously. Now, before we dive in, I want to mention a couple of important caveats um or things that I'll often be sweeping under the rug, but are important to know that are there.

So, in particular, let's imagine that when I collect my training samples, the features that I actually observe, I'm just going to call them Z for right now. So, these would be my observed features. And here I've got the labels Y. And imagine we're in more of a regression context where instead of just having positive or negative labels, we actually have scalar values. And imagine that my training samples look like this.

Right? So each of those red dots corresponds to a different let's say ZI Yi pair. Whoops. ZI Yi pair. Okay. So now within this framework I have two different options or yeah two different options. So option A is I would say that I'm going to define my feature vector X to simply be one-dimensional. It's just Z. So I'm in R1 or P is equal to 1. So in this case, yhat is going to be equal to a weight w1 * z. And then that's it.

And so now if I'm going to try to fit a line to this data, there we go. Following this model, then I'm going to be forced to go through the origin, right? because I have a linear equation here with a slope but no offset. So the alternative that I could consider is I could define my feature vector to have two components Z and just a constant one. So now I'm creating a two-dimensional feature vector. And if I do this and now I try to form a linear predictor, then what I'm saying is that y hat is going to be w1 * z plus w2.

And so if I try to find good weights that fit my data using this model, then I could find something that's a much better fit to my actual training data. And so often when we do this um we are going to assume that somebody's already made these decisions and formed our feature vectors for us. And often those feature vectors will have and often or if you're given just a data set and they those doesn't have these um constant offsets, it can be very helpful to add them.

Um and so when we talk about linear prediction, linear classification or linear regression here, we are going to just consistently not have any constant offsets that we're adding. We're going to just keep everything neat and clean like I've written on the left hand side of the board and assume that any constant offsets have already just been embedded into the feature vectors. Make sense? Okay. So it makes these models more flexible and capable of capturing other patterns um if we kind of augment our feature vectors this way.

Okay. Now the other thing that I want to be just really crystal clear about and sorry I just want to make sure that the camera is following me here. Good. is when we talk about classification, how we relate that to these linear models that we've been talking about. So when we talk about linear classification here, each one of the yis or all of the labels are going to be one of two values. Let's just say either + one or minus one.

So different than the plot that we had on the middle board when we were talking about a regression problem. So if we are given a weight vector W and then we compute um a prediction a predicted label let's say Y hat I as X I transpose W. In general, this inner product is not going to be either + one or minus one, right? It's just going to be some some number. And so when we want to do classification, we want to predict this positive or negative label.

And so what we do then is we form a classification rule. And by that what I mean is that um we are going to predict label + one oops + one if our linear or weighted sum xiranspose w is greater than or equal to zero and -1 one if xiranspose w is less than zero. And so graphically if we were to have on the horizontal axis axis all of the first features and on the vertical axis our um second feature. And then if we were to make a plot, what happened to my green?

Here we go of our training samples like we did last time. Then what we can do or plot on this diagram is a line like this. And what this line would correspond to is the set of all of the different feature vectors X. So that if I were to take the inner product of that feature vector and my weight vector, it gives me exactly zero. And so we call this our decision boundary because what we're going to do with this classification rule that I outlined is whenever we have a point X on this side of the decision boundary then we are going to declare label + one.

And whenever we have a point on this side of the decision boundary, we are going to declare label minus one. So we learn a weight vector. We can make predictions y hat just like we've been talking about all along. Those are going to be general scalar values. Sometimes it'll be 0.1, sometimes neg5, who knows? But if we really want to predict the label then we are going to see whether or not this inner product between the weight vector and the feature vector is positive or negative.

And the line dividing the positive and the negatives is this decision boundary. And so the location of that decision boundary or of that line is determined by the set of weights. And analogous to what we were doing in the regression setting, often we will have these feature vectors have a constant one inside them. That gives us flexibility about where exactly this line hits. Yes. >> Oh, right. Right. So the question is why did I choose that if the um if a point lies exactly on the decision boundary that I'm going to declare it a positive sample as opposed to a negative sample or does it matter?

Um the fact is we have to choose something. There's no exact right answer here and in practice data is just messy. So the chances of actually finding a sample that does lie exactly on the decision boundary. It just doesn't happen in practice very often. And so it's a little bit moot. Yeah, great question. Yeah. >> What are the axes? >> What are the axes on this plot? So this is the first feature of um whatever learning problem we have.

Let's just say in this face classification example, distance from eye corner to mouth corner. And this is the second feature. Let's say width of the mouth. Yeah. And I've got the little dot here just to indicate um that you know each each point here corresponds to a different eye or a different training sample. >> Yes. >> For for like the the first feature is Z and the second feature is one. How are we going to like are we just going to make X1= to Z and X2 just one? >> Okay.

So I think the question is when we were over here talking about regression, we were saying we could add this constant offset one here. Um what do we what does that mean in the context of classification? And so what I would say is that let's see um what you might like to do is to have a classification rule that says we are going to declare um plus one if xi transpose w is greater than some threshold tow and negative one otherwise.

Okay. So maybe you say, hey, I want to look at the weighted sum of my features and then I want to declare to be positive if it's above or below some threshold and I want to be able to choose that threshold. I don't necessarily want it to be zero, right? Um, so let's just think about what this means. Is this is this small or too big or too small? Sorry, is this too small? You guys can read this. Okay, so this is if we just are in our two-dimensional case, we're saying I've got w1 x i1 + w2 x i2 um being greater than or equal to tow.

And this is equivalent to saying w1 x i1 + w2 x i2 minus toao * 1 um being greater than or equal to zero. And so what I can do is I can just declare this negative toao as my third weight w3 and my one here as being part of my augmented feature vector. Let's just call it x i3. So I could take I could form an additional feature that is just a constant. And by doing that I'm essentially allowing myself to learn a good threshold here.

So it's giving me this additional offset that I can use um and learn along with all of the other weights. >> Yes. I'm sorry. Can you speak up? >> Like how can you draw like visualizing all >> Oh, how am I going to draw in more than two dimensions? No, it's not going to happen. I can only draw in two dimensions. Yeah. So, it's going to take a little bit of of imagination. Yeah. Yes. >> So, I'm unclear as to the benefits of introducing the uh like linear component because it doesn't seem like gives you more flexibility. line will always intersect the uh yaxis at one instead of at zero.

So is there some kind of like special number you're using beforehand because otherwise just seems like a shot in the dark with the >> So if we go back to this setting right uh we're we're assigning a weight to that constant offset one and that's going to determine what that offset is. So in this plot here, W2 is going to be where this classifier W2 is going to hit right there. So it is not going to be fixed at one. Great question.

Any other questions? Yes. >> So if you had I guess a feature vector with three distinct features, um would the decision boundary be a plane? Um, yes, that's right. So, ultimately what we're defining are like these hyperplanes that are splitting our overall P-dimensional space into two kind of half spaces. Yeah, that's right. All right. Excellent questions, everyone. All right. So, getting now to lease squares in the main event.

[Music] So in general the feature or I'm sorry the the labels that we actually observe for our training data are not going to be exactly equal to what we are able to predict with our weight vector. No matter how good our weight vector is, data has errors and noise. And so we aren't going to be able to get an exact fit in general. And so we are going to define the residual R I as simply being the difference between the training label and the predicted label.

So this is going to be y i minus y i hat or y i minus xirpose w. And then from there we can have a residual vector r which just has r1 r2 all the way to r n. So this is a vector in Rn. And so now what we are going to do when we talk about least squares is find the weight vector W to minimize the sum of the squared residuals. So I'm going to compute the sum from i = 1 to n. So the sum over all of my training samples. And for each one I am going to look at r i squared.

The sum of all of these squared residuals. And just to be really explicit about it, this residual depends on the weight vector. So each time I change my weight vector, I am changing the residual. So I'm trying to minimize over the set of weight vectors. this sum of squared residuals. And using some of the notation we talked about last time, this is the same as looking at the inner product of this residual vector with itself, which is the same as looking at the norm of that residual vector squared.

Okay, so this is what we refer to when we talk about least squares because we are trying to make the squared error as small as possible. And at this point, you might be asking yourself or me, why le squares? Why is this a good objective to consider? And later in the quarter, we're going to talk about alternatives to this lease squares formulation and why they have some advantages. But for now, this is a good starting point for a few reasons.

One, it magnifies really large errors. So, if I make a huge mistake, I'm going to pay more for it. Second, it makes the math easy. So we are going to be able to get very clean very simple expressions for the best set of weights. Um that is going to give us a lot of intuition, a lot of understanding. Um and it's one of the re that's this is one of the reasons why it's been used more than or and longer than any other method around.

Third and we're going to go into some detail about this today. It has a really nice geometric interpretation. And then finally, for those of you who are more interested in statistical aspects of these things, it coincides with modeling the true labels Y as being X times some true W star plus some noise or error and imagining that that error is coming from a Gaussian distribution. So we aren't going to spend a lot of time on the statistical modeling aspect of things in this particular course, but as you start taking more advanced courses after this, you might see this arise.

And so all of these are reasons why thinking about le squares can be extremely helpful. Are there any questions so far? Okay, so first of all, let me just tell you where we are headed. So a little spoiler here. The vector W that minimizes, the sum of the squared residuals is going to have the form W hat is equal to X transpose X inverse XRpose Y and the corresponding predicted labels can be written as yhat is equal to x * what which is the same as xrpose oop excuse as x * xrpose x inverse xrpose y.

And we are going to go into some detail about all of these things. First, I want to point out that when we're computing this weight vector, it's just a simple expression involving our training labels and our training feature vectors. We're just putting all of that together in one formula in order to get an expression for our um least squares weight vector. Now, what we're going to talk about today is when is this expression valid?

So, I already mentioned some assumptions that we're making on x. Um, and those are really critical to ensuring that this least squares predictor is a valid predictor. And we're going to talk about why that is and what happens when it's not true. We're also going to talk about what it means when we have xrpose x inverse like we haven't talked about matrix inverses yet. So we'll talk about that and we'll talk about why it arises here in the context of le squares prediction. and then finally we are going to derive this formula.

Okay, so that's basically the plan for the rest of the day and all of these things that we're going to cover are extremely important concepts. Um but once you have satisfied all the conditions you need to ensure that this is satisfied, you can take your favorite training data set and compute this le squares estimator without any trouble. Any questions so far? Yeah. >> Um okay. So we we we're using W's and not omegas here.

Um but this here is W hat, right? That's why these >> isn't that just >> is that zero everywhere? No, I don't think it's zero everywhere. >> Like we have like x wus. >> Oh, I'm sorry. This is an equal sign. Yeah, sorry about that. Okay. So, in order to start answering these questions, we're going to have to introduce a few additional concepts. And so, first we're going to talk about the span and range. And so in particular the span of a set of vectors and this is a general concept but I want to stay connected to our machine learning task.

And so I'm going to write these vectors as columns of our feature matrix X. And so if you remember this would be like x1 comma colon x2 comma colon all the way to x p or wait I think I've got my colons and numbers mixed up. Apologize colon comma 1 colon comma 2 all the way to colon comma p. So these would be for instance the p columns of my feature matrix. But in general, if I've got a set of vectors of P different vectors here and all of them are in Rn, so they're all vectors of real numbers with n different elements.

Um, the span is the set of vectors that can be written I say set of vectors [Music] as a weighted sum of my p original vectors. So of the x colon j's. So in other words, the span of x1 through x colon p is equal to the set of all vectors and I'm just going to call these vectors y tilda and they are in the same dimension as my original vectors. So they are in Rn and I can write that y tilda vector as being a weighted sum of these xjs.

So as a weight 1 * x cola 1 all the way to weight p * x cola p for some weights w1 w2 through wp. I'm going to give you a few examples in just a second that'll make this really crystal clear. But let me just define the range real quick here. So if my matrix X has got columns X col 1 X2 all the way to X colon P. So if I've got this matrix and it's n by p, then I say the range of x is the span of its columns. Okay, so range and span are intimately connected like this.

All right. So, let's work through a couple of concrete examples to really make that clear because I think when I write it like that, it sounds a little abstract, but the idea is quite simple and as we're going to see in a minute, intimately connected to this notion of these squares. Okay. So in particular in this example [Music] imagine that my x1 is equal to 1 0 0 and x col 2 is equal to 0 1 0. So now if we wanted to compute the span of these vectors, it would be any vector that could be written as a weighted sum of these two vectors.

So for instance this or more specifically I guess the span of x1 and x colon 2 is all vectors of the form um alpha or let's just put weight one weight two zero. So any vector that looks like this for any set of weights w1 and w2 is in the span of these two vectors because I can write this vector here as a weighted sum of these two vectors. Make sense? [Music] Okay. So this now is going to lead us directly to understanding the geometry of le squares.

And I'm going to need a full board for that. [Music] And we are going to do this in the setting where we have um p is equal to two. So we have two features and n is equal to three. We've got three different training samples. And so now we are going to want a weight vector that's two-dimensional. And we are going to be working in the span of the columns of X. So I'm going to have [Music] this plane here in 3D space. So this plane is equal to the span of X1 and X2.

So here's my origin. And here in the plane I've got x1 and and also in the plane I've got this point x colon 2. Okay. So all of that is sort of in lying in this one plane in green here. So recall what these two vectors correspond to. This vector corresponds to the value of feature one across all n of our samples. And this corresponds to the value of feature 2 across all n of our samples. So these are both vectors in 3D space because we've got three different samples.

Okay? But there's only two vectors like this because we only have two features. And so when we think about their span, that's going to be a set of different vectors that all lie in this plane here. Now, simultaneously, we also have our labels, our label vector y that's coming from our training data. So, it's also in this 3D space, but it is not necessarily in this span. And these labels that we've observed might not be exactly a weighted sum of these features, right?

There's going to be some difference. And now what we want to do is we want to try to find a set of weights that is going to minimize these squared residuals. And the way this is going to look is the following. So I can imagine trying to find the point in this plane in this span of these two vectors that's as close as possible to y. So this is going to be my y hat the thing that I am predicting with my weight vector. And remember that the difference between y and y hat was our vector of residuals.

This is the thing that we want to make as small as possible. And so this yhat now is inside this span. I lost my green. But y hat is inside this span that we've drawn here in this green plane because it is going to be a weighted sum of our different features. So just by the way that we've constructed this problem where we said we're trying to make a linear prediction, we said that our predicted labels will be weighted sums of our features.

And so by definition, this vector of predicted labels has to be in this span. And so what we want to do is we want to find this vector of predicted labels that's going to be as close as possible to the labels of our training data by minimizing these sum of this sum of squared residuals. Now you could imagine some other point in this plane. Let's just call it y tilda. It's also a weighted sum of these feature vectors and it's got some distance from our true or training labels Y.

And so we're going to call that vector the difference between Y and Y tilda R tilda. So this is just Y minus Y tilda. And so what I would argue is that well what we want is we want the y hat that has the shortest or the smallest sum of squared residuals or the shortest length of this vector. And if we were to look at the shortest length vector versus any other vector, we know that this one here, the one that we care about is going to be perpendicular to this span.

So in particular, if I were to look if I were to form a triangle here by looking at this thinking about this y hat as being a vector emanating from the origin, I'm going to have a right triangle here. I'm going to have because in order for this y to be as close as possible to y hat, I have to be kind of orthogonal to this span. and any other vector that I might consider corresponding to any other set of weights is not going to form a right angle because it's farther away.

Right? Maybe it's a little bit hard to draw this, but imagine for instance that the tip of my chalk is equal to this point y um underlying or the vector of labels y and I want to find the point on the board that's as close as possible to the chalk. So, I want to draw the shortest possible line that's going to be perpendicular to the board. If I were to choose some other line like this that's not perpendicular to the board, it's going to be a longer line, right?

Okay. So, same concept here. We're choosing a point y that's perpendicular to this span um to this plane and line uh sorry, the residual vector is perpendicular to the plane. yhat is inside the plane, but we've got this right triangle here, and that's what's going to minimize these sum of squared residuals. Okay, so let me just write some of that a little bit more explicitly. So in particular r our residual vector is orthogonal to this span of x colon 1 and x2 because Um let's see any other arta would have longer length [Music] and by that I mean that if we were to look at this uklid norm squared of r tilda.

It would necessarily have to be bigger than the uklidian norm of r. I want to add a little bit more to that. [Music] So in particular, [Music] I'm going to think about the vector corresponding to the difference between yhat and y tilda. And I'm going to call this vector d. And I want to note that since we said that this vector R is perpendicular to this span, then this here is also a right angle. And so if we were to consider some y tilda some other weighted sum of the features that does not minimize the sum of squared residuals.

So I'll just write this as does not minim or does not have minimum uh r till the norm in that case um or let me see let me put it the other way that is not perpendicular [Music] to the span of the features. Then the corresponding R tilda length, the sum of the squared residuals associated with this Y tilda would have to be bigger than the least square solution. And we can see this by going back to our figure and thinking about this triangle here R and D and this blue line here uh R tilda.

So in particular by the Pythagorean theorem when we've got this right triangle we know that the length of the um hypotenuse squared our tilda is going to be the sum of the squared lengths of the two sides. which is R and D. And so if this d vector is not equal to zero, so if we've got some y tilda and it's not the same thing as our y hat that we care about, then this tells us that the length of our tilda has to be greater than the length of our le squares estimate r.

So all of this is to say that our yhat has to be this point in the span a weighted sum of our features. And in order for it to be the point as close as possible to y, it has to have a residual vector that's orthogonal to this span or perpendicular to this span. Anything that's not orthogonal is going to have a longer length residual vector which corresponds to a bigger sum of squared errors. Okay. So now that we've established that it has to be orthogonal, we can use this to derive the equation that I had on the board earlier.

But before I move on, do does anybody have any questions? >> Yeah. Okay. So I think the question is um why is this y hat in the span of the two vectors x1 and x2. Is that right? >> Could it be possible for y to be on? Um yes, it is possible for the observed training label vector to lie in this span. It's possible. In general, it doesn't actually happen. It's not very common. If it happened, then you whatever y hat that we want that's closest to y would just be y, right?

The problem is really trivial then, right? But um so so we generally assume that it's different. But um yeah and and in practice it is different. It does not lie exactly in the span. >> Yeah. >> So for these like yhat and y tilda they're all vectors. >> Yes. Yhat y tilda um the rs the d they're all vectors. And I'm trying to be consistent and writing an underline under the vector. I'm also trying to stay consistent with the textbook.

And it it does um as much as I I love the textbook, when it's referring to vectors that are columns of matrices, it does not underline them. So I apologize that that's a tiny bit confusing, but for any other vector, I'm trying to be consistent about underlining it so you recognize it. Did you have a question? Yeah, for the second >> sorry >> for the second vector. >> Oh, so if I understand correctly, you're asking if um I should write this without the squares. that follow. >> Oh.

Oh, sorry. Yes. Typo here. Yeah. Thank you very much. Yeah. Sorry. Yeah. >> So, how do we visualize the n= to 3D here? Is it just going to be like a different plane, a different span? >> My 3D drawing skills are not great, >> but we are thinking about this being a drawing of 3D space. And so um this span is like a 2D plane in 3D space and y is a vector in that 3D space that's not in that plane. That's what I'm trying to represent with the drawing if that makes sense. >> But I'm I'm a little bit confused about like so the n here is three and we're just looking at like I I can be like one or two or three, right? >> The x i.

Yeah. So, we're kind of masking the individual samples and just treating them as different dimensions uh or different coordinates in this three-dimensional space. So, when I say that y is a 3D vector in or you know I'm trying to represent it as a 3D vector here. Um it's got y1 y2 y3 those are the elements of it and that's determines where this point is. Is that make sense? Okay. Yes. So like here we have like number of like samples equals to three but like features only equal to true and then why isn't on the plane but doesn't that like imply like if we have like n items we need n features >> um so the question is if we've got n samples do we need n features and um the answer is no and let's just think about an example where for instance We want to classify whether a photo is smiling using features like mouth width and distance from eye to mouth.

So there we've got two features that I just listed. And I might have thousands of different images. And I don't need to have thousands of features for this to make sense. You can also maybe just recall the picture that we had earlier where we had a whole bunch of positive examples and a whole bunch of negative examples and we tried to draw a decision boundary. Um so this is an example where each of these pluses and minuses is a different sample but we only have two features so they do not need to be equal. >> But like in this illustration like the y vector is a threedimensional vector but we are only on like a two plane.

So we can never approach the y with x. >> So um yes the y vector is in 3D space and our columns of our feature matrix x are spanning in this case just a 2D plane because we only have two features. And so when we form our y hat as a weighted sum of these vectors of features it also is going to have to be in that plane. So it will never exactly in this diagram it'll never exactly be y but our goal is to find a set of weights so that yhat is as close as possible to y even though it can never be exactly y. >> Yeah. >> So y is the distance between the two types of vectors.

So r here is the residual vector and it the length of that corresponds to the distance or the squared. Yeah, the distance between um our training labels and our predicted labels. Okay, great. Excellent questions. So now that we've kind of built this up and established that this notion of orthogonality is critical, we can use it to describe or to derive our least squares formula [Music] or at least get part of the way there.

Okay. So in particular we said that W hat is the argument or the vector that minimizes the sum over all of our training samples of the residual RA I associated with W. uh squared. Okay. So I'm going to use a little shortand here and I'm going to call this the arg min over w. So this is saying I'm going to search over all W's and try to minimize the sum from i = 1 to n r i^2 w and what I want to be output is not the minimum value but the argument w that gives me that minimum value.

So that's what argument means here. We're going to see that a lot as we talk about different optimization problems. And as I said before, this is equivalent to saying we want the arg min over w of the inner product of r with itself, which is the same as the arg min over w of the length of r 2. Okay. Now what we just established through this geometric perspective that whatever W corresponding or W hat corresponding to Y hat is going to give us a residual vector that's perpendicular to this span.

So W hat is going to lead to a residual vector R that is perpendicular or orthogonal to the span of the X colon J's of all of our columns here. And so when we talk about orthogonal vectors, two vectors V and U are orthogonal if the inner product is equal to zero. Okay, so we're going to use that and we're going to say well what does this perpendicular or this orthogonality property that we derived from this geometric perspective imply about w.

So in particular what we have is that if we were to look at the inner product of this residual vector with any one of the vectors in this span it has to be equal to zero. So if I were to take X colon J one of these vector of features. So the vector of the J feature across all of my end training samples. And if I were to take the inner product of that with this residual vector, then that has to be equal to zero for all J for all of my different P features.

So equivalently I can just stack all this together and I can say that X transpose R is equal to zero. Right? So I've got all of these XJs together in one big feature matrix X. And so this orthogonality principle is telling us that XRpose * R has to be equal to zero. And now I'm going to just fill in our definition of r. So I've got xrpose and now we can remember that r this residual was the difference between our training vector or a vector of training labels y and our predicted labels and our predicted labels are going to be x what.

So this product must be equal to zero. in order for our yhat or this this residual vector to be orthogonal to the span. And so this tells us that W hat has to satisfy that Xrpose Y is equal to XRPOSEX W. So if we were to work backwards, right? If we had or if we have a w hat that satisfies this equation, then when we look at this equation, it will indeed equal zero. And that will imply that the corresponding residual vector is in fact orthogonal to this span of feature vectors.

So by using this geometric principle um or geometric perspective and the understanding that our residual vector is as small as possible as short as possible when it is orthogonal to this span. we derive this um equation or this system of linear equations that tells us what has to be true that that W hat has to satisfy in order to minimize the sum of squared errors. >> Yeah. >> How did the ranspose rule? >> Um yeah, so I did Let's just write a few things down for that.

[Music] So we said that our transpose x colon j is equal to zero and this is true for all of the j's from 1 to p. So then what I could do is I could keep this ranspose and then I can just stack all of these XJs together. So I could have x1 x2 all the way to x colon p. And so if I do this multiplication, this is going to give me ranspose x1, ranspose x colon 2, all the way to ranspose x colon p. Okay, so now I know that all of these need to be equal to zero.

So this second line is maybe uh and then going back to the previous um board this is like saying ranspose x. So the first line saying that for each individual vector of features it has to be orthogonal to r and then I just concatenated all of it to get one much simpler expression. And then um what we had was ranspose x is equal to zero. And then what I did is I took the transpose of both sides of this expression. So this is a vector of zeros and a vector of rs.

And so if I were to take rpose x and transpose that and that's the same as zero transpose here. So in this case, let's just talk about dimensions real quick. Um, let's see. R was in R N and X was in R N by P. And so this zero vector here is P by or 1 by P. Okay. So now when I try the transpose of this this is equal this is a p by1 vector of zeros and when I take the transpose of a product like this what I do is I take the transpose of each individual factor and I reverse the order.

So I'm going to take xrpose and then I'm going to have ranspose transpose which is simply xrpose r and then that's equal to zero. Yeah, thanks for asking. Any other questions? Yes. >> Um yes, sorry this is yes thank you. These are all zero vectors. um except for this one which really is a scalar. >> Yes. >> So when we're talking about a span here does it mean like the span has to be like um of the columns of x no more no less like all the columns. >> That's right.

So if we recall um we said when we write that yhat is equal to x w um I think in maybe last week I said well remember that x is x1 x 2 through x p * w. So this is the same as saying that yhat is w1 * x1 plus w2 x2 all the way to wp x p. And so what I have here now is that y hat is a weighted sum of the columns of x. And if we remember our definition of span of these columns, then it corresponds to any vector that can be written as a weighted sum of those columns.

And so y hat now just by the way we've constructed our linear predictor is in that span. >> Yeah. Yes. >> So on that second line, shouldn't those not be zero vectors? It's already a vector. >> Um here. >> Yeah. Oh, did I I'm sorry. >> You're right. I apologize. Yep. Thank you very much. These are scalers. In this line, we have vectors. This line scalers. Thank you very much.