Introduction to the adjoint method transcript

so here let's let me introduce this idea which is we call the adjoint method by giving you a very simple but actually very hot person to solve so what if I what do you what if I have a optimization problem like this I want to minimize okay I want to minimize C transpose times X where X is the solution to the parameterize the link equation equation a ax equal to B which is a function of s and here I want to minimize over the parameter space of s so imagine my ax equal to B of s is a structural equation you are solving and s is parameters that parameterize is the the the system set up the design okay and I imagine I want to minimize a certain deformation which is a which is described by the known vector C so here the problem C is known the matrix a is known the function B is known the thing that is unknown is s and you want to figure out what kind of s is going to give me the minimum C transpose X now here's the challenge can you solve the problem without ever without ever solving the system ax equal to B all right I mean a natural way to think of solving this problem is for example I'm just gonna try a milling different esses and further all of these are mailing different asses I'm gonna compute respect respective X and for each X I can compute in C transpose X the objective function I'm just gonna pick from this meeting solutions the one that works the best right that's a brute force way of calculating of solving the optimization problem but the challenge here is what can I do if I'm not allowed to compute the solution ax for any of these asses is there still a way to do it since there are just the three of you you guys are free to discuss the line of thought is correct so so maybe let me do it a little bit more rigorously over here the line of thought is exactly correct so instead of not allowed to solve this but I can I can actually write X right whatever ax I have is actually a inverse times P of s right and if I'm able to substitute this into the objective function then C transpose X is actually equal to C transpose a inverse times P of s so this is my X which depends on what s is but I'm not obligated to compute this one first I can use the Association rule to compute this first right so this is a where the somebody comes from if I compute a transpose B first I have to do these every time I change s but if I compute C transpose x inverse first I can just do that once all right which means if I can if I call this if I call this X hat let's say which means that well let me call this X hat transpose then this is basically saying I pre compute X X hat transpose equal to C transpose times a inverse or if I write this in a little bit more if I transpose both sides and move the a inverse to the left hand side this is actually a minus transpose oh sorry not a - transpose a transpose times a so it is X hat a transpose times X hat equal to C so this is equivalent to that right if I if I take if I can if I multiply both sides by the inverse of a transpose and transpose both sides I get these so if I solve this equation first this is actually by the way called the add to an equation in this case then the objective function can be written as X hat transpose times B then from now on I don't really have to optimize the original function I can optimize this function is that right I just evaluate B again and again without ever having to solve any linear equations again so what's the idea we get from this example is that the order of the operations is very important it can be very important in determining the cost to the computation cost of your optimization algorithm the backpropagation method I'm going to describe next it's actually a very much extension of this idea instead of instead of looking an expression and start evaluating the expression in the usual way which is computed your pass then compute them the next logically next one we we can actually reverse the whole operation and start from the left side start from the side that is close to the objective function and the propagate in a backwards way right so that's the idea of the back propagation