title: Lie-Access Neural Turing Machines
abstract: 
    External neural memory structures have recently become a popular tool for
    algorithmic deep learning
    (Graves et al. 2014; Weston et al. 2014).  These models
    generally utilize differentiable versions of traditional discrete
    memory-access structures (random access, stacks, tapes) to provide
    the storage necessary for computational tasks.  In
    this work, we argue that these neural memory systems lack specific
    structure important for relative indexing, and propose an
    alternative model, Lie-access memory, that is explicitly designed
    for the neural setting.  In this paradigm, memory is accessed using
    a continuous head in a key-space manifold. The head is moved via Lie
    group actions, such as shifts or rotations, generated by a
    controller, and memory access is performed by linear smoothing in
    key space. We argue that Lie groups provide a natural generalization
    of discrete memory structures, such as Turing machines, as they
    provide inverse and identity operators while maintaining
    differentiability. To experiment with this approach, we implement
    a simplified Lie-access neural Turing machine (LANTM) with
    different Lie groups.  We find that this approach is able to perform
    well on a range of algorithmic tasks.
authors: Greg Yang Alexander Rush
hIndex: Alexander Rush 43