Multiple vector–Jacobian products are cheap Original Research Article

All methods for solving linear or nonlinear systems of equations require the evaluation of residual vectors, most algorithms are based on Jacobian–vector products and many schemes involve vector–Jacobian products. Moreover, while only some exact Newton methods form and factor the Jacobian explicitly, some information about the size of its elements or the structure of its spectrum appears to be indispensable for preconditioning purposes. In this paper we will examine what statements can be made about the relative costs of obtaining these various quantities for a linear or nonlinear mapping between Euclidean spaces. Our only assumption is that this vector function is evaluated by a computer program as a composition of arithmetic operations and univariate algebraic or transcendental functions such as the square root and the exponential. Complete Jacobians are in general an order of magnitude more expensive to obtain than the underlying residual. The complexity ratio between these two mathematical objects depends on the sparsity of the Jacobian and other structural properties of the vector function in question. Analytical values for Jacobian–vector products are never much more expensive than residuals, which is also the case for their approximation by divided differences. In fact, sometimes such directional derivatives are significantly cheaper, especially if evaluated as a bundle. These observations are relevant for Krylov or block-Krylov methods, irrespective of whether the problem is linear or not. Iterative equation solvers that consistently reduce some fixed norm of the residual over successive steps tend to involve vector–Jacobian products. Partly, because evaluating these row-vectors is generally assumed to be quite expensive or at least troublesome for the user considerable effort has gone into the development of transpose-free variants that require only Jacobian–vector products. The good news of this article is that vector–Jacobian products can in principle be obtained at the same operations count as Jacobian–vector products. We will also consider the calculation of Hessian–vector products, which can be obtained directly from the code for evaluating the underlying scalar-valued function.