title: Cached Transformers: Improving Transformers with Differentiable Memory
    Cache
summary: This work introduces a new Transformer model called Cached Transformer, which
  uses Gated Recurrent Cached (GRC) attention to extend the self-attention
  mechanism with a differentiable memory cache of tokens. GRC attention enables
  attending to both past and current tokens, increasing the receptive field of
  attention and allowing for exploring long-range dependencies. By utilizing a
  recurrent gating unit to continuously update the cache, our model achieves
  significant advancements in six language and vision tasks, including
  language modeling, machine translation, ListOPs, image classification, object
  detection, and instance segmentation. Furthermore, our approach surpasses
  previous memory-based techniques in tasks such as language modeling and
  displays the ability to be applied to a broader range of situations.
paper_authors._id: 6583b44583a9e1460c68f5e7
paper_authors.name: Zhaoyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-21T14:05:47.894Z
paper_authors._id: 658409ceca19ccf6d9989add
paper_authors.avatarUrl: /avatars/3ac1dbd25e52435185babdeb3da28875.svg
paper_authors.fullname: Zhaoyang Zhang
paper_authors.type: user
paper_authors.user: ZyZcuhk
paper_authors._id: 6583b44583a9e1460c68f5e8
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:52:08.198Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 6583b44583a9e1460c68f5e9
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:52:14.862Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 6583b44583a9e1460c68f5ea
paper_authors.name: Xiaogang Wang
paper_authors._id: 6583b44583a9e1460c68f5eb
paper_authors.name: Jinwei Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T08:52:30.811Z
paper_authors._id: 647e8118770c299e56fc2bc8
paper_authors.avatarUrl: /avatars/adf80f3473dda42450148789ae5c208f.svg
paper_authors.fullname: Jinwei Gu
paper_authors.type: user
paper_authors.user: jwgu
paper_authors._id: 6583b44583a9e1460c68f5ec
paper_authors.name: Ping Luo
paper_title: Cached Transformers: Improving Transformers with Differentiable Memory
    Cache
paper_summary: This work introduces a new Transformer model called Cached Transformer, which
  uses Gated Recurrent Cached (GRC) attention to extend the self-attention
  mechanism with a differentiable memory cache of tokens. GRC attention enables
  attending to both past and current tokens, increasing the receptive field of
  attention and allowing for exploring long-range dependencies. By utilizing a
  recurrent gating unit to continuously update the cache, our model achieves
  significant advancements in six language and vision tasks, including
  language modeling, machine translation, ListOPs, image classification, object
  detection, and instance segmentation. Furthermore, our approach surpasses
  previous memory-based techniques in tasks such as language modeling and
  displays the ability to be applied to a broader range of situations.
paper_ai_summary: A Cached Transformer model with Gated Recurrent Cached attention achieves significant advancements in language and vision tasks by extending self-attention with a differentiable memory cache.
paper_ai_keywords: Transformer Gated Recurrent Cached (GRC) attention self-attention differentiable memory cache recurrent gating unit language modeling machine translation ListOPs image classification object detection instance segmentation