title: Efficiently applying attention to sequential data with the Recurrent Discounted Attention unit
abstract: Recurrent Neural Networks architectures excel at processing sequences by
  modelling dependencies over different timescales. The recently introduced
  Recurrent Weighted Average (RWA) unit captures long term dependencies
  far better than an LSTM on several challenging tasks. The RWA achieves
  this by applying attention to each input and computing a weighted average
  over the full history of its computations. Unfortunately, the RWA cannot
  change the attention it has assigned to previous timesteps, and so struggles
  with carrying out consecutive tasks or tasks with changing requirements.
  We present the Recurrent Discounted Attention (RDA) unit that builds on
  the RWA by additionally allowing the discounting of the past.
  We empirically compare our model to RWA, LSTM and GRU units on
  several challenging tasks. On tasks with a single output the RWA, RDA and
  GRU units learn much quicker than the LSTM and with better performance.
  On the multiple sequence copy task our RDA unit learns the task three
  times as quickly as the LSTM or GRU units while the RWA fails to learn at
  all. On the Wikipedia character prediction task the LSTM performs best
  but it followed closely by our RDA unit. Overall our RDA unit performs
  well and is sample efficient on a large variety of sequence tasks.
authors: Brendan Maginnis Pierre Richemond
hIndex: Pierre Richemond 4