A universal learning rule that minimizes well-formed cost functions

Author

Mora-Jiménez, Inma ; Cid-Sueiro, Jesús

Author_Institution

Dept. of Signal Theor. & Commun., Univ. Carlos III de Madrid, Leganes-Madrid, Spain

Volume

16

Issue

4

fYear

2005

fDate

7/1/2005 12:00:00 AM

Firstpage

810

Lastpage

820

Abstract

In this paper, we analyze stochastic gradient learning rules for posterior probability estimation using networks with a single layer of weights and a general nonlinear activation function. We provide necessary and sufficient conditions on the learning rules and the activation function to obtain probability estimates. Also, we extend the concept of well-formed cost function, proposed by Wittner and Denker, to multiclass problems, and we provide theoretical results showing the advantages of this kind of objective functions.

Keywords

gradient methods; learning (artificial intelligence); maximum likelihood estimation; neural nets; probability; cost function minimization; neural network; nonlinear activation function; posterior probability estimation; stochastic gradient learning rule; universal learning rule; Amplitude modulation; Artificial neural networks; Bayesian methods; Cost function; Entropy; Logistics; Medical diagnosis; Optimization methods; Stochastic processes; Sufficient conditions; Generalized soft perceptron (GSP); stochastic learning rule; strict sense Bayesian (SSB) cost function; well-formed cost function; Algorithms; Cluster Analysis; Computer Simulation; Computing Methodologies; Decision Support Techniques; Models, Biological; Models, Statistical; Neural Networks (Computer); Numerical Analysis, Computer-Assisted; Pattern Recognition, Automated; Stochastic Processes;

fLanguage

English

Journal_Title

Neural Networks, IEEE Transactions on

Publisher

ieee

ISSN

1045-9227

Type

jour

DOI

10.1109/TNN.2005.849839

Filename

1461424