Bounds on approximate steepest descent for likelihood maximization in exponential families

Author

Cesa-Bianchi, Nico ; Krogh, Anders ; Warmuth, Manfred K.

Author_Institution

California Univ., Santa Cruz, CA, USA

Volume

40

Issue

4

fYear

1994

fDate

7/1/1994 12:00:00 AM

Firstpage

1215

Lastpage

1218

Abstract

An approximate steepest descent strategy is described, converging in families of regular exponential densities to maximum likelihood estimates of density functions. These density estimates are also obtained by an application of the principle of minimum relative entropy subject to empirical constraints. We prove tight bounds on the increase of the log-likelihood at each iteration of our strategy for families of exponential densities whose log-densities are spanned by a set of bounded basis functions

Keywords

convergence of numerical methods; entropy; information theory; iterative methods; maximum likelihood estimation; numerical analysis; approximate steepest descent; bounded basis functions; convergence; density estimates; density functions; empirical constraints; exponential densities; exponential families; iteration; likelihood maximization; log-densities; log-likelihood; maximum likelihood estimates; minimum relative entropy estimation; regular exponential densities; tight bounds; Computer science; Councils; Density measurement; Entropy; Equations; Extraterrestrial measurements; Iterative methods; Maximum likelihood estimation; Upper bound;

fLanguage

English

Journal_Title

Information Theory, IEEE Transactions on

Publisher

ieee

ISSN

0018-9448

Type

jour

DOI

10.1109/18.335953

Filename

335953