Abstract
It is commonly assumed that calculating third order information is too expensive for most applications. But we show that the directional derivative of the Hessian (\(D^3 f(x)\cdot d\)) can be calculated at a cost proportional to that of a state-of-the-art method for calculating the Hessian matrix. We do this by first presenting a simple procedure for designing high order reverse methods and applying it to deduce several methods including a reverse method that calculates \(D^3f(x)\cdot d\). We have implemented this method taking into account symmetry and sparsity, and successfully calculated this derivative for functions with a million variables. These results indicate that the use of third order information in a general nonlinear solver, such as Halley–Chebyshev methods, could be a practical alternative to Newton’s method. Furthermore, high-order sensitivity information is used in methods for robust aerodynamic design. An efficient high-order differentiation tool could facilitate the use of similar methods in the design of other mechanical structures.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Specifically \(P^T\) would be \(\Psi ^1\) and \(\Phi ^{i}\) would be \(\Psi ^{i+1}\).
As checked May 28th, 2013.
The bandwidth of matrix \(M=(m_{ij})\) is the maximum value of \(2|i-j|+1\) such that \(m_{ij}\ne 0\).
On the function scon1dls, both methods generate different fill-ins that are five orders of magnitude smaller than the remaining entries.
References
Griewank, A., Walther, A.: Evaluating derivatives, 2nd edn. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (2008)
Gebremedhin, A.H., Manne, F., Pothen, A.: What color is your Jacobian? Graph coloring for computing derivatives. SIAM Rev. 47(4), 629–705 (2005)
Griewank, A., Naumann, U.: Accumulating Jacobians as chained sparse matrix products. Math. Progr. 95(3), 555–571 (2003)
Gower, R.M., Mello, M.P.: A new framework for the computation of Hessians. Optim. Methods Softw. 27(2), 251–273 (2012)
Gebremedhin, A.H., Tarafdar, A., Pothen, A., Walther, A.: Efficient computation of sparse Hessians using coloring and automatic differentiation. INFORMS J. Comput. 21(2), 209–223 (2009)
Gutiérrez, J.M., Hernández, M.A.: A family of Chebyshev–Halley type methods in Banach spaces. Bull. Aust. Math. Soc. 55(01), 113–130 (1997)
Amat, S., Busquier, S.: Third-order iterative methods under Kantorovich conditions. J. Math. Anal. Appl. 336(1), 243–261 (2007)
Wang, X., Kou, J.: Semilocal convergence and R-order for modified Chebyshev–Halley methods. J. Numer. Algorithms 64(1),105–126 (2013)
Ezquerro, J.A., Hern, M.A.: New Kantorovich-type conditions for Halley’s method. Appl. Numer. Anal. Comput. Math. 77(1), 70–77 (2005)
Xu, X., Ling, Y.: Semilocal convergence for Halley’s method under weak Lipschitz condition. Appl. Math. Comput. 215(8), 3057–3067 (2009)
Susanto, H., Karjanto, N.: Newtons methods basins of attraction revisited. Appl. Math. Comput. 215(3), 1084–1090 (2009)
Yau, L., Ben-Israel, A.: The Newton and Halley methods for complex roots. Am. Math. Mon. 105(9), 806–818 (1998)
Gundersen, G., Steihaug, T.: Sparsity in higher order methods for unconstrained optimization. Optim. Methods Softw. 27(2), 275–294 (2012)
Gundersen, G., Steihaug, T.: On diagonally structured problems in unconstrained optimization using an inexact super Halley method. J. Comput. Appl. Math. 236(15), 3685–3695 (2012)
Gundersen, G., Steihaug, T.: On large-scale unconstrained optimization problems and higher order methods. Optim. Methods Softw. 25(3), 337–358 (2010)
Papadimitriou, D.I., Giannakoglou, K.C.: Robust design in aerodynamics using third-order sensitivity analysis based on discrete adjoint. Application to quasi-1D flows. Int. J. Numer. Methods Fluids 69, 691–709 (2012)
Papdimitriou, D.I., Giannakoglou, K.C.: Third-order sensitivity analysis for robust aerodynamic design using continuous adjoint. Int. J. Numer. Methods Fluids 71, 652–670 (2013)
Ozaki, I., Kimura, F., Berz, M.: Higher-order sensitivity analysis of finite element method by automatic differentiation. Comput. Mech. 16(4), 223–234 (1995)
Ederington, L.H., Guan, W.: Higher order greeks. J. Deriv. 14(3), 7–34 (2007)
Kariwala, V.: Automatic differentiation-based quadrature method of moments for solving population balance equations. AIChE J. 58(3), 842–854 (2012)
Corliss, G.F.F., Griewank, A., Henneberger, P.: High-order stiff ODE solvers via automatic differentiation and rational prediction. In: Vulkov, L., Waśniewski, J., Yalamov, P. (eds.) Lecture notes in computer science, pp. 114–125. Springer, Berlin, Heidelberg (1997)
Guckenheimer, J., Meloon, B.: Computing periodic orbits and their bifurcations with automatic differentiation. SIAM J. Sci. Comput. 22(3), 951–985 (2000)
Griewank, A., Walther, A., Utke, J.: Evaluating higher derivative tensors by forward propagation of univariate Taylor series. Math. Comput. 69(231), 1117–1130 (2000)
Naumann, U.: The art of differentiating computer programs: An introduction to algorithmic differentiation. Number 24 in Software, Environments, and Tools. SIAM, Philadelphia (2012)
Neidinger, R.D.: An efficient method for the numerical evaluation of partial derivatives of arbitrary order. ACM Trans. Math. Softw. 18(2), 159–173 (1992)
Fraenkel, L.E.: Formulae for high derivatives of composite functions. Math. Proc. Camb. Philos. Soc. 83(02), 159 (1978)
Griewank, A., Juedes, D., Utke, J.: ADOL-C, a package for the automatic differentiation of algorithms written in C/C++. ACM Trans. Math. Softw. 22(2), 131–167 (1996)
Bongartz, I., Conn, A.R., Gould, N., Toint, P.L.: CUTE: constrained and unconstrained testing environment. ACM Trans. Math. Softw. 21(1), 123–160 (1995)
Hock, W., Schittkowski, K.: Test examples for nonlinear programming codes. J. Optim. Theory Appl. 30(1), 127–129 (1980)
Luksan, L., Vlcek, J.: Test problems for unconstrained optimization. Technical Report 897, Academy of Sciences of the Czech Republic (2003)
Abate, J., Bischof, C., Roh, L., Carle, A.: Algorithms and design for a second-order automatic differentiation module. In: Proceedings of the 1997 International Symposium on Symbolic and Agebraic Computation (ACM), pp. 149–155, New York (1997)
Walther, A., Griewank, A.: Advantages of binomial checkpointing for memory-reduced adjoint calculations. In: Feistauer, M., Dolejší, V., Knobloch, P., Najzar, K. (eds.) Numerical Mathematics and Advanced Applications, pp. 834–843. Springer, Berlin, Heidelberg (2004). ISBN: 978-3-642-62288-5. doi:10.1007/978-3-642-18775-9_82
Sternberg, J., Griewank, A.: Reduction of storage requirement by checkpointing for time-dependent optimal control problems in ODEs. In: Norris, B., Bücker, M., Corliss, G., Hovland, P., Naumann, U. (eds.) Automatic Differentiation: Applications, Theory, and Implementations, 0, pp. 99–110. Springer, 1 edition (2006)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gower, R.M., Gower, A.L. Higher-order reverse automatic differentiation with emphasis on the third-order. Math. Program. 155, 81–103 (2016). https://doi.org/10.1007/s10107-014-0827-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-014-0827-4
Keywords
- Automatic differentiation
- High-order methods
- Tensors vector products
- Hessian matrix
- Sensitivity analysis