Complete Memory Structures for Approximating Nonlinear Discrete Time Mappings


       Bryan W. Stiles, Irwin W. Sandberg, and Joydeep Ghosh
       Department of Electrical and Computer Engineering 
       The University of Texas at Austin

                     ABSTRACT
This paper introduces a general structure that is capable of approximating 
input-output maps of nonlinear discrete time systems. The structure is 
comprised of two stages, a dynamical stage followed by memoryless nonlinear 
stage. An approximation theorem is presented which shows that certain
structures of this form are capable of modeling a large class of nonlinear 
discrete time systems. In particular, we introduce the concept of  a ``complete
memory.'' A structure with a complete memory dynamical stage and a
sufficiently powerful memoryless stage is shown to be capable of approximating
arbitrarily well a wide class of continuous, causal, time-invariant, 
approximately finite memory mappings between discrete time signals. In fact it
is shown to have the same universal approximation capability as time delay 
neural networks (TDNNs) or focused gamma networks. Furthermore any uniformly 
bounded, time-invariant, causal memory structure has such an approximation 
capability if and only if it is a complete memory. Therefore the memory stages
of both TDNNs and focused gamma networks are complete memories. Both of these
networks have linear memory stages. An example of a nonlinear complete memory
dynamical stage is also presented. The proposed complete memory structure
provides a theoretical basis for designing a wide variety of artificial neural
networks for nonlinear spatio-temporal processing.