Some limitations of linear memory architectures for signal processing

Certain neural network structures with a linear “memory” stage followed by a nonlinear memoryless stage are commonly used for signal processing. Two examples of such structures are the time delay neural network and the focused gamma network. These structures can approximate arbitrarily well a wide range of mappings between discrete time systems. However, in order to achieve this capability, the dimensionality of the output (state vector) of the linear memory stage is allowed to be arbitrarily large. In practice the dimensionality of the state vector must be limited due to finite resources and in order to reduce problems while training the memoryless stage arising from “the curse of dimensionality”. We discuss how such a limitation effects the range of functions which can be approximated by the structure. Further, it is proven that given any tolerance and any limit on the dimensionality of the state vector, there are computationally simple and useful functions which cannot be approximated to the given tolerance by any linear memory structure (including TDNNs and the focused gamma network) which conforms to the prescribed limit on the state vector dimensionality. The existence of such functions provides a rationale for examining structures with nonlinear memory