Evolution-based hessian approximation for hybrid numerical optimization methods

Abstract

In this dissertation, Hessian approximation algorithms are proposed to estimate the search direction of the quasi-Newton methods for solving optimization problems of continuous parameters. The proposed algorithms are quite different from other well-known quasi-Newton methods such as Symmetric Rank-One(SRO), Davidon-Fletcher-Powell(DFP) and Broyden-Fletcher-Goldfarb-Shanno(BFGS) in that the Hessian matrix is not calculated from the gradient information but from the function values directly. The proposed algorithms are designed for a class of hybrid methods that combine evolutionary search with the gradient-based methods of quasi-Newton type. The function values calculated for the evolutionary search are used for estimation of the Hessian matrix (or its inverse) as well as the gradient vector. The Hessian matrix is recursively corrected so that its curvature matches the curvature information found from the fitness values. For this purpose, two different update methods, one called batch update and the other sequential update, are proposed. Furthermore, convergence properties of repetitive application of the proposed update schemes are investigated, both analytically and numerically. Since the estimation process of the Hessian matrix is independent of that of the gradient vector, more reliable Hessian estimation with a sparse population is possible compared with the previous methods based upon the classical quasi-Newton methods. Numerical experiments show that the proposed algorithms are very competitive with regards to state-of-the-art evolutionary algorithms for continuous optimization problems.