Lecture 5 Fall 2025: Subspaces and Bases transcript

All right, welcome back everybody. So today we are going to talk about subspaces and bases. Now these are topics that sound initially a little bit abstract. They're closely connected though to things we've already been discussing in the context of le squares. And by understanding these, we're going to lead directly into the singular value decomposition and principal components analysis, which are really central to all of machine learning.

And so these are really kind of vital and central topics. So to start off, let's just recall our um geometric picture of lease squares. And so in particular what we had we had drawn this sort of hyper plane. So, we imagined that we were in um a 3D space and we had this 2D um plane and when we drew this we said that this plane corresponded to taking different columns of our feature matrix. So if we recall our feature matrix X had rows let's say X1 transpose X2 transpose corresponding to um our different samples our different training samples.

Each column in this matrix corresponded to a different feature. And so in this picture, these green arrows corresponded to say the first column and the second column. So each one of these vectors is a vector in 3D space because we have n equals three different samples in this example and we are in a 2D plane. P is equal to two because we have two different features. And remember that what we were doing is we were looking for weighted sums of those features.

So in particular, we had a label vector y of three different labels. And what we tried to do was to find a point in this plane that was as close as possible to y. And so this point here was going to be um x w or it was going to be equal to weight 1 * x * the first feature plus weight 2 * the second feature. All right. And so within this context that we talked about quite a bit, we call this green plane now a subspace and in particular it is spanned by the columns of X.

And so it's the same concept as before. And actually, let me just add as well. Um, so this point that we're talking about here, let's just call this Y hat. Y hat we said is the projection of y onto the subspace. Okay. So the same basic concepts that we've already been talking about. But today we're going to talk a little bit in more detail about these basic concepts of a subspace span projection and along the way we'll also talk about different ways we can represent subspaces including bases.

So we're going to do this a little bit more formally and add to what we've already seen. So in particular, let's start off with the notion of a subspace. And to start, let's consider um all points x in our n. So, we've got all the different points in an n-dimensional space. A subspace is a subset of these points [Music] that basically have some special properties. And in particular for a subspace um if we were to consider any two points in the subspace and let me just call them here X and Z.

And if we were to consider any two weights, any two scalar weights, let's just call them alpha and beta. Then the weighted sum of x and z with weights alpha and beta also has to be in the subspace. So alpha * x plus beta * z is also in the subspace. Okay. So maybe this sounds a little bit abstract I think understandably. And so what we're going to do is we are going to work through just a ton of different examples to make this clear.

And so for the first one, let's consider being in n= 3D space. So we've got three different coordinate axes. And then imagine that we have So yeah, just maybe this is the x1 direction, x2 direction, x3 direction. I'll do this like that. And now imagine that I've got two points. hard to see exactly where they are in 3D, but I'm going to call these points X and Z. And so what I can do is I can form a subspace by thinking about all weighted sums of these points.

So any point on this line would correspond to a point of the form alpha X plus beta Z. So the only thing that's different for different points on this subspace are the values of those weights alpha and beta. Right? Right here alpha is equal to 0 and beta is equal to 1. Here alpha is equal to 1 and beta is equal to zero. Notably the origin is always in a subspace. If you have a subspace it has it contains the origin because anytime you choose weights to all be zero then the weighted sum is zero.

And so the zero vector always always has to be in the subspace. Yes. >> That's right. I always use um or try to always use an underline for a vector. So when I write this, I'm saying that these are vectors in this n-dimensional space. Um so in this picture, they are supposed to be vectors in 3D space. And so this subspace now we're going to talk more about the dimension of a subspace but in this example that dimension of the subspace is one corresponding to it just being a 1D line.

Yes. translate >> right this so the question is when I drew this I I very conveniently drew the line right through the origin will that always be the case and so in this particular example I am forming the subspace by considering all weighted combinations of x's and z's and so since the weights could include zero it would have to go through the origin. Um if I were to take so so this is distinct from let's just say having a different point let's say over here x I don't know tilda right and then just like drawing a line connecting them okay this is a line but this is not a subspace okay so this is not a subspace I'll give another example differentiating creating sort of a a plane from subspace later on.

But this is these are not these points along this yellow line do not correspond to all weighted sums of Z and X tilda. And you can tell that immediately because the origin is not in that because if I were to compute the sum of Z * 0 plus X tilda time 0, it has to be the origin. So anytime you see a line or a plane or a hyper plane, it doesn't go through the origin. You know immediately that it cannot be a subspace. It's something else.

Great point. In addition, I want to point out that subspaces are not bounded. It's not like it just sort of stops at some point. It keeps on going forever because I can make these weights in this case alpha and beta as large as I want infinitely large. So there's never any sort of like bound on the extent of a subspace. So let's just write these two things down. The origin or the zero vector is always in a subspace and subspaces are not bounded.

Okay. Are there any other question? Yeah. Oh, so this is changing the third coordinate. So when I write um alpha x + beta z what I mean here is alpha * x1 x2 x3 plus beta * z1 z2 z3 and so this is alpha x1 + beta z1 alpha x2 + beta z2 and alpha x3 plus beta z3. And so every point on this white line has this form. And so when I vary alpha and beta, I am varying this third coordinate. Now, if my original two points X and Z had just zero values for that third coordinate, then um no matter what weights I put on it, these points would all have zero values there.

So, we'll talk about that example exactly in a second. Um but so far in this example, I'm not assuming anything about the value of the third coordinate. Great. Okay. Yes. >> In the case of X is the subs subspace just a plane defined by X, Z, and Z. >> Um, in this particular example, the subspace is onedimensional because um I'm really just uh well, in yeah, in this drawing, this is one-dimensional. More generally, if we're talking about the span of two different vectors, then we would have a two-dimensional subspace.

Um, so it's really just in this example that we've got a one-dimensional subspace. I was just trying to visualize something for you. Okay, great. Was there another question? Thought I saw another hand. Okay, so let's look at another example and maybe that'll also help clear some things up. So I'm often going to use um a curly s to denote a subspace like that. Just so we're all on the same page there. So let's have a subspace s correspond to all the points x that are in a 3D space where um the first coordinate x1 is equal to the second coordinate x2 is equal to the negative of the third coordinate x3.

So this is in fact a subspace. Um and in particular we could think about um like we could have uh a single point in this subspace. So I don't know let's just say z is equal to 1 1. So this point is in our subspace and if we were to consider any weight alpha then alpha * z is going to have the form alpha alpha negative alpha and that satisfies this constraint where the first two coordinates are equal and the third coordinate is the opposite of the first two and that also is going to be in this subspace.

And so we can also say that this subspace is equal to the span of the vector 1 1. So remember when we talked about the span earlier we said the span of a set of vectors let's say v_sub_1 v2 through vp was equal to the set of vectors that could be written as a weighted sum of the v's. This okay so this is another example of a subspace corresponds exactly to the span of a single vector. It includes the origin. it does not have um any bound on its extent.

Okay, for another example, again, let's work in three dimensions because we're going to try to make a drawing. And we're going to consider a subspace that's equal to the set of all points x in 3D where x1 is equal to x2. Okay, so this is more con less constrained than our previous example because in our previous example, we also put constraints on what the third coordinate should be. Here we have no constraints on that third coordinate.

So we can try to make a drawing of this. So I am in my 3D space. So here is my x1 direction and my x2 direction and my x3 direction. And then drawing this is a little hairy but imagine first we're in the horizontal plane and this dashed line corresponds to all the points in the horizontal plane where x1 and x2 are equal to each other. So we've got that line and now we're saying we have no constraints on what the third coordinate should be.

And so the thing you should be picturing here is like a big I'm I'm drawing lines but remember that these things are not bounded. So I'm just trying to illustrate the fact that we've got a plane here, a vertical plane. It goes infinitely far in all directions. And that plane is sort of intersecting this x1 = x2 line inside the horizontal plane, but going infinitely far vertically and infinitely far along that line. So this is a 2D subspace. in a 3D space.

Any questions about this example? [Music] Okay. So now we could consider another example. Again we're going to work in 3D space and our subspace in this case will be the set of all points X in 3D. So that the third coordinate X3 is equal to zero. And other than that, we are placing no constraints whatsoever on the coordinates x2 and x1. So again, we've got our 3D space here. Here's x1, x2, and x3. And what does the subspace look like in this example?

Yeah, >> right. Just the horizontal plane. this x1 x2 plane it does go into all the quadrants. I'm just sort of trying to illustrate it's the horizontal plane here by illustrating it as being bounded even though it goes infinitely far in all directions. So this also is a a subspace. And then of course there is the kind of major example that we've talked about a lot in this class. The example corresponding to our le squares machine learning problem.

So, so we are given vectors of features and these correspond to columns of our feature matrix. So since they're columns of a matrix, we'll write colon comma 1 cola 2 colon comma 3. And these are all in an nd dimensional space. So it corresponds to the first, second, and third feature. And each vector corresponds to the value of that feature across all n of our training samples. So now an example of a subspace is equal to the span of these points x1 x2 and x 3.

And if we recall our definition of the span and what we did with le squares, this is the set of all points, let's just say y tilda that are in Rn that can be written as [Music] a weighted sum of those three different vectors. So we can write it as w1 x1 w2 x2 and w3 x3 for some values of w1 w2 and w3. So any point that I can write as a weighted sum of these x's would correspond to a um point in this subspace. Okay. Any questions about any of these examples so far? >> Yeah. >> In this particular case, yes, this was a line.

And I think this is probably what the first question at the beginning was was alluding to. In this example, I carefully chose this x and y so that even though I'm looking at the span of two vectors, they are on a line that goes through the origin. And so together they form a 1D2 subspace instead of a 2D subspace. That's generally not the case. >> Wouldn't you only need one of those two vectors though to form that same? >> That's right.

So we're going to talk about how we can represent subspaces in just a second. So in this particular example, this subspace is the span of X and Z. But as you're absolutely correctly pointing out, we could just write it as the span of X. We don't actually need the Z. It's just a 1D subspace. So we only really need to have one vector to represent it. And so representing it in terms of X and Z is inefficient. And that's what we're going to get to when we talk about bases.

Um, nevertheless, this subspace here is the span of X and Z. It might not be the most efficient way to write it, but it is that is a correct statement. And what I was really just trying to illustrate with this example is how we can think about weighted sums of any two points in the subspaces also being in the subspace. >> Yes. Is there a difference between subspace and >> um when we take the span of a collection of vectors that forms a subspace and given a subspace we can represent it as the span of a set of vectors.

So in that sense they're intimately connected. Um but for instance in this example here that we started with right this is a subspace. Now I can do some additional work to represent that as the span of some vectors. But before I do that this is just a geometric object right it is just a subspace. It's not yet a span of vectors. Um so they're closely tied together. You're 100% right. But this subspace object exists even without me writing it as the span of a set of vectors.

Yeah, great questions. Any other questions? Yes. >> Were we talking about like space then? >> So, okay. Um, just for the recording, the question is, you know, we've we've certainly looked at things like predictors that do not go through the origin. So just as an a simple example, right? Imagine that this horizontal axis is x and the vertical axis is y and then we find a predictor maybe that looks something like this.

It's not going through the origin. So this is a a line. It's telling us for any new value of x, what's our predicted value of y. But this is not a subspace in and of itself. It's it's just a line that's kind of telling us the functional mapping you know so that for any new x you can say what the corresponding y prediction would be. So that's a distinct notion from this notion of a subspace where we really are trying to think about our um um predictions as being uh let me put it this way.

This corresponds to a particular set of weights, right? We've got the weight that tells us what the slope is and the weight that tells us what the offset is. But when we are using our training data to find this line to find that set of weights, what we're doing is we are searching over all possible sets of weights which would give us the set of all possible different lines there and we're choosing the one that best fits our training data.

And so that set of all possible different lines that correspond to weighted sums of our um of our features that is a subspace. So the decision boundary might not be might not go through the origin or our regression prediction line here might not go through the origin. But as we are searching for our least squares weights, we are considering um predicted label vectors that do lie in a subspace. So it's a a distinction there. >> Yes. >> Okay.

Yeah, great question. So let's just make this concrete. So let's let's say here that we are in this sort of 1D example where we only observe feature Z and then we form our uh feature vector let's say X I by taking ZI and then concatenating a one. And so then if we were to do that, our feature matrix is going to look like Z1, Z2 all the way through ZN and then 1 one one. So this is what I am calling the first column of X.

So in the earlier example, I wrote this as X colon comma 1 for the first column of X. And this here is my second column of X. And that second column is just a vector of all ones. And so now when we think about our um geometric view of le squares, then we've got these two different vectors. So now here we've got this x comma 1. So this is just this vector corresponding to Z1 through ZN. And then I've got this other vector X comma 2.

And this is just a vector of all ones. And so now what we're doing when we do these squares is we take our actual vector of labels Y and we project down. And this is where we get um let's just say our yhat that's going to be equal to w1 * x comma 1 + w2 x comma 2. And in this particular case, this particular example we're playing with, this is like saying that my predicted label for the E sample is going to be W1 * Z + W2 * 1.

And so if we go back to our picture over here, then this intercept here is simply W2. and w1 is telling me what this slope is the rise over run. And so that line is not a subspace. But when we consider all the possible yhat vectors that we could have selected through any possible set of weights, then that is going to correspond to vectors in this green plane in this 2D subspace. So that's the subspace and then this is the particular lease squares predictor that we found within that subspace. >> Yes. >> But the problem is if the final like decision boundary is not in the subspace like how can we find that something not in the subspace from? >> So we are choosing something within the subspace.

We are searching over predictors that correspond to weighted sums of vectors. So we absolutely are constraining our search to vectors within this subspace. >> Yeah. >> But then why is the final decision boundary not? Does that mean the decision boundary isn't like part of the search in the sub? So each point in this subspace does correspond to a set of weights. And then separately we can say what is the decision boundary corresponding to that set of weights or in the regression context what does the regression function look like for that set of weights.

But that decision boundary that we've drawn for a particular weight vector or this regression line for a particular weight vector that is not a subspace. It's just defined by a set of weights that correspond to a point in the subspace. So the subspace is corresponds to the set of all possible prediction or or predicted label vectors on our training samples. And that's distinct from the decision boundary or the regression line that tells us the that's corresponding to one special point in this subspace that's the least squares point close as close as possible to our training labels.

Yes. >> So is it accurate to say that zero is on the green plane but not necessarily that idea. >> Okay. So absolutely yes, zero is in this green plane and it's it's right there at the middle and that's why I've drawn all the vectors as kind of emanating from it. So like a little more abstractly, people might just think about these as as points and I'm just drawing them as as arrows because I think that helps with the geometric perspective.

But yeah, so the origin is here and I'm thinking about all of these points in terms of their where they are relative to the origin. Yeah. Yes. >> So the definition of subspace we have an equation alpha plus beta that character sub. >> So when we talked about our definition of the subspace we had the alpha and beta weights. Those correspond exactly to the w1 and w2 here. So this would be like the alpha. This would be like the beta in this example.

So I've got these two points and these two vectors here are like the x and z in our definition of the of a subspace. And when we define the span of this sub um of these two vectors which is itself a subspace we're thinking about all weighted combinations of them. Yeah. Any other questions? Okay, these are fantastic questions. I have one more example and then we are going to talk about um bases. So let's start by recalling the Netflix problem or data.

It's not a problem yet. So we are going to have this matrix X and if you can if you recall we said each row would correspond to a particular movie and each column would correspond to a particular user or customer. And so if we looked at a particular entry of this matrix, it tells us how much that customer likes that movie. And that's what this matrix is is representing. And then what we did is we said imagine that we could write this matrix as the product of two other matrices.

So we had a matrix U and then we had another matrix V. And so what we said in this example is that a um column of U might correspond to a representative movie watcher. So for instance, this column might correspond to the ratings of each movie for a romance lover and another column might correspond to the ratings of each movie by a horror lover. And we said this matrix V then would tell us um for each individual customer how aligned they are with the romance lover or the horror lover, the action lover, etc.

So we talked about this before. So in this setting we had for instance n different movies and p different customers and we imagined that we only needed our different representative movie watchers in order to represent this matrix. And so in this context there is basically a subspace and in particular we say that each users ratings. And by that I mean a each column of this ratings matrix X is a weighted sum of the columns of U. and therefore it lies in the subspace equal to the span of the columns of U.

[Music] So I'm not really giving you any new insight here as this is just one example of a subspace. When you have a matrix that can be factored this way that can be written as the product of two other matrices the way I have then we can say that this these columns of U span a subspace and each column of X each individual customer's ratings lies within that span or lies within that subspace. >> Yes. >> Here's the rank.

If we were to make the number of columns here as small as possible so that we still have this equality x is equal to u * v then r would be the rank. It is possible though to find matrices u and v with more columns and rows than strictly necessary. So R is not so so it's possible in general to take a matrix X and write it as the product of two other matrices with this inner dimension R where R is actually bigger than the rank but if you've made R as small as possible then it is the rank and in fact as I mentioned earlier we're building towards the singular value decomposition and that's going to give us exactly that that rank.

So, we're we're headed right towards where you're thinking. Did I see another hand? No. Okay. [Music] All right. So, now I want to get back to one of the earlier questions, which is how do we actually represent a subspace? And in particular, I gave you a little toy example um where I had two points that were on a 1D subspace and somebody said, "Well, do I really need to represent the subspace with both of those points?" And the answer was no.

So, let's just talk about that in a little bit more detail. Um, [Music] okay. So the simplest way that we might represent or one of the simplest ways we might represent a uh subspace would be as the span of a set of vectors. So this is fine, it's accurate, but as we heard earlier, this could be inefficient, right? Because we might have more vectors than we really need. Um, and so this could be a lot of redundancy here.

[Music] So then we could be a little bit more strict here and say, well, we're going to represent our subspace as the span of a set of linearly independent vectors. [Music] Okay, so same idea, but now we're saying we only want linearly independent vectors. That's going to eliminate the redundancy. This is even better. And this is called a subspace basis. Okay. Okay. So when we talk about a basis for the subspace, all we mean is a set of vectors that are linearly independent.

They're not redundant with each other that are going to span our subspace. But the third option is to be even more constrained and talk about the span of a set of orthonormal vectors. And this is called an orthonormal subspace basis. Okay. So let's talk about what orthonormal means. So if I have a set of vectors [Music] So if I have a subspace that's equal to the span of a set of vectors and I'm going to write them as U1 U2 through what did I use in the notes here?

You are um the UI's are orthonormal if two things are true. First, if I were to take the inner product of a pair of them, ui and u.j and then that inner product is either going to be one or zero. It's going to be one if I is equal to j and zero otherwise. Okay. So this second condition that says if I take any two different vectors and I compute their inner product and it's zero this means that they are orthogonal and this first condition that if I take the inner product of any one vector with itself so remember this is the same as the length or the norm of ui that's also having to be equal to one, this corresponds to normal or normalized.

So when we say that it is orthonormal, we mean that every single vector here has length one. It's not like some of the vectors are really short and some are really long. They all have the same length and that length is one. And they are orthogonal to one another. Um, and so not only are they linearly independent, but they are perpendicular. And this turns out to be very convenient for a lot of different machine learning problems.

Having an orthonormal basis for a subspace turns out to give us just makes it simplifies a lot of math. It can simplify a lot of computation and so it's very helpful to have. [Music] Now once we've got a basis then we can represent that basis with a matrix. So let's define that real quick. [Music] Here we go. [Music] So I could form a matrix U that's going to have columns U1 U2 all the way through UR. And so if um uh you uh if we've got an orthonormal basis Then we know that each column of U has norm one and each pair of columns is orthogonal. and so let's see in particular if we were to compute uranspose time u if we were to think about that product well we talked before about what that would look like the First entry would be U1 transpose U1.

The second entry would be uh U1 transpose U2 all the way down to U1 transpose UR and then we would have U2 transpose U1 U2 transpose U2 etc. So if we have orthonormal columns in this matrix then what can we say about these inner products? Sorry. Right. We're going to have an identity matrix because anytime we are on the diagonal we're taking the inner product of one of these vectors with itself. And we said when it's orthonormal that has to be equal to one.

So along the diagonal we've got ones. And anywhere where we're off the diagonal we're taking the inner product of two different basis vectors. They're orthogonal. So that inner product has to be equal to zero. So any entry off the diagonal is going to be zero valued. So in other words, uranspose U is equal to the identity matrix when we have an orthogonal basis matrix orthonormal basis matrix. Another property of this orthonormal basis matrix is that it is length preserving.

[Music] And by that what I mean is if I were to take any vector V that is um in this rd dimensional space and then I were to look at the product of U * V. Then first of all what is this? What this is giving me is a weighted sum of the columns of U. But in particular, I can say, well, what is the length or the norm of that product? And what I'm going to do here is I'm going to square that. And we'll think about what that squared length or norm is.

And this is equal to UV transpose time UV. And now I'm going to take the transpose of that first product in parentheses. So I get vranspose uranspose u v. And now what can I do? >> Yes. >> Upose is the >> exactly right. We just showed that uranspose u is the identity. So now I have vranspose v and this is just the length of v^2. So u is length preserving in the sense that if I take any vector v and multiply u * v the length of that product is the same as the length of v as measured by this uklidian norm.

Yes. >> Oh thanks very much. So when I write we we've talked about this before. when I write V like this um then I implicitly mean it with the two and what I mean in particular is the sum um let's just say from J= 1 to R of V J^ squared and then at the very end I do 1/2 so when I so whenever I don't write anything here this is what I mean but we can define more generally what people call an LP norm, which is a little confusing because we use P in this class for something else.

Um, but this is commonly referred to as the LP norm. So, I'm going to leave it here. And what this means is I'm going to compute the sum from J= 1 to R of the VJS to the P power and then at the end take the 1 over P root. And so there are variety of settings where using a different P is helpful. Very common example, one particularly widely used example is P is equal to 1. And in fact we might talk about that later in the quarter.

Um and so sometimes people will specify what they mean and I just do it by habit. But whenever I don't write it, I do mean just this same two norm that we've been talking about in class. Yeah. Any other questions? Okay. So, let's work out an example. And earlier we talked about we talked about the subspace in 3D. So, of all points x that are in R3 where we had um the third coordinate being equal to zero. So this was a subspace and now what I want to do is think about representing that subspace using a basis.

And so all vectors x that are in this subspace have the form um I don't know some alpha and beta and zero. So we don't have any constraints on the first or the second coordinate but the third coordinate has to be equal to zero. And so we could actually write this as for instance alpha * the vector 1 0 0 plus beta * the vector 0 1 0. So any vector in this subspace can be written as a weighted sum of these two vectors.

And so I'm just going to call this first vector u1 and this second vector u2. And now what I have is an orthonormal basis which is just U1 and U2. And I have a basis matrix U that looks like 1 0 0 1 0 0. Okay, there's one other thing that I meant to mention earlier and forgot, but we're going to do it right now. If I have a set of vectors U1, U2 through UR and they form an orthonormal basis for S. Then r the number of the uis is equal to the subspace dimension.

And if we were to have um uh a matrix uh this base uh yeah maybe let me say it this way and the rank of the basis matrix U is equal to R. So we were talking a little bit earlier about the rank of a matrix and um this is intimately related to a subspace dimension. So if I've got a subspace and I find a basis for it, an orthonormal basis, then the number of basis vectors that I need corresponds to the dimension of that subspace.

And if I for take all those basis vectors and form a matrix, the rank of that matrix is equal to that number of basis vectors. So in our examples for instance in this one where our subspace corresponded to the horizontal plane then overall we're thinking about points in 3D but that subspace is just a 2D plane and our basis matrix here only has two different columns or two different basis vectors representing the fact that it is a 2D basis sorry 2D subspace.

All right. Yes. [Music] >> R numbers. Um what I was trying to say here is that um if you have a basis for your subspace and that basis has is vectors u1 u2 through ur then r which is the number of these basis vectors is the subspace dimension. That's all I meant. [Music] Okay, so one quick example, [Music] one subspace is of course just the set of all r dimension or n dimensional vectors that is a subspace not a very interesting subspace but if you were to take any two points in this subspace and compute their weighted sum they're also in the subspace satisfies our definition um and one thing I want to point out is that um how did I phrase this um a subspace [Music] n Rn cannot have more than n basis vectors.

[Music] Okay, just a helpful sanity check when you're working out problems. But now what I want to do is I want to talk about a special basis for Rn that I have already sort of hinted at. So we are going to let Ei correspond to the vector that has that's all zeros except for a one in the E location. This is like saying it is the e column of an identity matrix that is n by n. Okay. So in our earlier example over here when we were forming our first basis for this problem then our first basis vector u1 this is an example of this special basis.

So we could call this E1 and U2 is E2. Okay. So this is just a special example of basis vectors that's worth mentioning because it's used quite widely. Um so these are called canonical vectors. [Music] Um, and so any veh any vector in Rn [Music] can be written in terms of them. So for instance, you could have a vector x that's equal to 2.5, pi, and 10. And this is simply 2.5 * e1 +<unk> * e2 + 10 * e3. So this is a nice basis in the sense that if I hand you a vector, you can write its basis.

You can write that in terms of the basis trivially. Don't even have to do any complicated math. Okay. So now what I'd like to do is to compare or well first of all emphasize the fact that given a subspace there's not a unique basis. There's multiple different even orthonormal bases that we could use to represent that subspace. And this is important because we are going to look at two concrete mechanisms for taking a bunch of data points and identifying a basis for the subspace that they lie in.

And they're going to give us different bases. And so in this example, we are going to have an initial basis that corresponds to our canonical vectors. So we are going to be in 3D again. [Music] So this is a basis [Music] for R3. So if we make a picture here. So here's my x1 coordinate. Here's my x2 coordinate. And here is my x3 coordinate. So in this setting, my first basis vector corresponding to the first column of u is going to be this.

This is u1. And my second basis vector right here. This is U2 and my third basis vector is U3. So now if you were to give me any point in this 3D space, I can describe where it is to you in terms of a weighted sum of these coordinates, which is exactly how you represent points in terms of coordinates than you have since junior high. But we could also consider and and this is a nice one. This is an orthonormal basis. This is an example where uranspose u is the identity matrix and in this case we are the this space that we are working in is full dimensional right we're in a 3D space and the subspace is also 3D so in this case actually u uranspose is also equal to the identity but we could also have an alternative basis this.

And I'm going to call this one utilda just to emphasize the fact that it's different. And this could be 1 over the<unk> 2 1 over the<unk> 2 0 1 over the<unk> 2 - 1 over the<unk> 2 0 and then 0 0 1. So let's just think about what that looks like. So I'm going to use orange for this. So over here now I am having my first basis vector be along the x1 = x2 line. So this is my utilda1 basis vector. And then I've got another u2 tilda and these are perpendicular to one another.

And then my third basis vector u3 is exactly the same as before. U tilda 3. These are both completely valid orthoronormal bases for exactly the same space. So in this case we also have utild transpose utild is equal to the identity matrix. In this special case we also have utilda utildpose is equal to the identity matrix. So we've got all the same nice properties. We just have two different bases, two different ways of representing where points might lie within this space.

There's no real deeper point here except that there can be multiple different bases for a space, which is important to realize when we start talking about different ways to compute a basis for a subspace spanned by a collection of data points. Now, in order to do this, we are going to have to use a concept that let's see, we've got about seven minutes left. Okay. So, I think what I want to do with our remaining time is work through another example that's related to the example that we had Sorry.

Looks like my camera is off and I don't know how long it's been off. Sorry guys. I don't know what to say. That sucks. Okay. So in this example we are going to consider the set of all points x that are in three dimensions where the third coordinate x3 is equal to the value five. And so we are going to call this set of points curly x. Is this a subspace? >> Yeah. >> Exactly right. This is not a subspace because the zero vector is not inside this space because every vector in this space has the third coordinate equal to 5.

So it's impossible to have a point in this space with the third coordinate equal to zero. Now we can also consider two vectors V1 which is and V2 which is -3 4 5. Both of these vectors v1 and v2 are inside this space x right they both have that third coordinate equal to five. But now what we can do is we can think about the span of vub1 and v2 and this is a subspace and so let's just think about what that subspace would look like.

So that span is going to have the form every vector in that is going to have the form alpha * v1 plus beta * v2. And if we just spell that out kind of pedantically, we'll have 2 alpha - 3 beta and alpha + 4 beta and 5 alpha + 5 beta. And so if I were to choose alpha and beta equal to zero, then the origin is inside this span. So notably even though these two vectors are inside X and X is not a subspace when we look at their span we do have a subspace and we are going to get vectors in that subspace that are not always in X.

That third coordinate here depending on what I choose for alpha and beta might not be five. In fact it might be zero which is extremely important but it really might be a variety of different numbers. And so we can try to draw try try to draw a picture of this. [Music] So here's my x1 direction. Here's my x2 direction. And here's my x3 direction. The printed lecture notes online are going to be much better at this drawing.

And so first we're going to have this set X curly X. And so if this here is five, what we're talking about is sort of a big horizontal plane here. So this here is X. It's a nice big plane, but it is not a subspace because it does not go through the origin. And then what I can do is I can look at my two different vectors V1 and V2 that are both in this plane. So here's one of them and here's another. So this Whoops. So here is V1 and here is V2.

And now if I think about the span of those vectors, I can think about trying to basically fit a plane to those vectors that is going so that every point on both of these vectors is in that plane. And so what I get is sort of this other plane here. This is my subspace corresponding to the span of v_sub_1 and v2. It contains the origin. It contains v_sub_1 and v2. But every possible weighted combination of v_sub1 and v2 as well.

So these are two different two-dimensional spaces. One corresponds to a subspace in orange. one absolutely is not a subspace in green. Both contain the two vectors v_sub_1 and v2. They are in that intersection between the two. Um but the span of those vectors is not necessarily this original set x that we started with. Okay. So next time on Thursday we are going to talk about the first way of computing an orthonormal basis from data using Graham Schmidt orthogonalization and then next week we start with the singular value decomposition and principal components analysis.

Thanks very much. >> Thank you.