Backpropagation through time with fixed memory size requirements

Author

Principe, Jose C. ; Kuo, Jyh-Ming ; de Vries, Bert

Author_Institution

Comput. NeuroEng. Lab., Florida Univ., Gainesville, FL, USA

fYear

1993

fDate

6-9 Sep 1993

Firstpage

207

Lastpage

215

Abstract

A generalization of the backpropagation through time (BPTT) algorithm is presented, which, under reasonable assumptions, can lead to fixed memory size requirements. The idea is to model BPTT as the storage of activations and errors in tapped delay lines, and then generalize the tap delay line to a gamma memory. Since the depth of the gamma filter is T=K/μ, where K is the filter order and μ a scalar that can be varied between 0<μ<1, it is possible to achieve depth T with a fixed filter order K. The accuracy of the gradient computation is tested in an example

Keywords

backpropagation; content-addressable storage; generalisation (artificial intelligence); neural nets; backpropagation through time; delay lines; errors; fixed memory size; gamma filter; gamma memory; generalization; gradient computation; learning; neural nets; Backpropagation algorithms; Biological neural networks; Boundary conditions; Convolution; Delay lines; Filters; Laboratories; Neural engineering; Real time systems; Testing;

fLanguage

English

Publisher

ieee

Conference_Titel

Neural Networks for Processing [1993] III. Proceedings of the 1993 IEEE-SP Workshop

Conference_Location

Linthicum Heights, MD

Print_ISBN

0-7803-0928-6

Type

conf

DOI

10.1109/NNSP.1993.471868

Filename

471868