Vikramaditya Reddy Jakkula Traditional supervised neural network
trainers have deviated little from the fundamental back
propagation algorithm popularized in 1986 by Rumelhart, Hinton,
and Williams. Typically, the training process begins with the
collection of a fixed database of input and output vectors. The
operator then adjusts additional parameters such as network
architecture, learning rate, momentum, and annealing noise,
based upon their past experience in network training. Optimizing
the network’s generalization capacity usually involves either
experiments with various hidden layer architectures or similar
automated investigations using genetic algorithms. Along with
these often-complex procedural issues, usable networks generally
lack flexibility, beginning at the level of the individual
processing unit. Normally, the user finds him or her confined to
a limited range of unit activation functions, usually including
linear, linear threshold, and sigmoidal analytical forms whose
partial derivatives with