title: DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales
abstract: Extracting temporal relationships over a range of scales is a hallmark of
  human perception and cognition---and thus it is a critical feature of machine
  learning applied to real-world problems.  Neural networks are either plagued
  by the exploding/vanishing gradient problem in recurrent neural networks
  (RNNs) or must adjust their parameters to learn the relevant time scales
  (e.g., in LSTMs). This paper introduces DeepSITH, a deep network comprising
  biologically-inspired Scale-Invariant Temporal History (SITH) modules in
  series with dense connections between layers. Each SITH module is simply a
  set of time cells coding what happened when with a geometrically-spaced set of
  time lags.  The dense connections between layers change the definition of what
  from one layer to the next.  The geometric series of time lags implies that
  the network codes time on a logarithmic scale, enabling DeepSITH network to
  learn problems requiring memory over a wide range of time scales. We compare
  DeepSITH to LSTMs and other recent RNNs on several time series prediction and
  decoding tasks. DeepSITH achieves results comparable to state-of-the-art
  performance on these problems and continues to perform well even as the delays
  are increased.
  
authors: Brandon G Jacques Zoran Tiganj Marc Howard Per B Sederberg
hIndex: Marc Howard 36