Quadratic Optimization via Conjugate Directions and Projection Matrices

The idea behind the well known conjugate gradient procedure is to solve a series of one-dimensional optimization problems along direction vectors that are a function of both the current gradient vector and the previous search vector. The search vectors are sequentially generated allowing the optimization process to move along one direction at a time while making these vectors Q orthogonal, where Q is the nxn weighting matrix from the quadratic objective function. In this work, a method is presented in which the search directions are all initially fixed as the columns of the Q matrix. It is then shown that for this choice, the Gram-Schmidt orthogonalization process can be used to locate the extremum in n steps. It is also shown that the original search directions become conjugate directions after these n steps. The net result is a new and efficient conjugate direction method.