title: TCNCA: Temporal Convolution Network with Chunked Attention for Scalable
    Sequence Processing
summary: MEGA is a recent transformer-based architecture, which utilizes a linear
  recurrent operator whose parallel computation, based on the FFT, scales as
  O(LlogL), with L being the sequence length. We build upon their approach by
  replacing the linear recurrence with a special temporal convolutional network
  which permits larger receptive field size with shallower networks, and reduces
  the computational complexity to O(L). The resulting model is called TCNCA, a
  Temporal Convolutional Network with Chunked Attention. We evaluate TCNCA on
  EnWik8 language modeling, long-range-arena (LRA) sequence classification, as
  well as a synthetic reasoning benchmark associative recall. On EnWik8, TCNCA
  outperforms MEGA, reaching a lower loss with 1.37times/1.24times faster
  forward/backward pass during training. The dilated convolutions used in TCNCA
  are consistently and significantly faster operations than the FFT-based
  parallelized recurrence in GPUs, making them a scalable candidate for handling
  very large sequence lengths: they are up to 7.07times/2.86times faster in
  the forward/backward pass for sequences up to 131k. Further on LRA, TCNCA
  achieves, on average, 1.28times speed-up during inference with similar
  accuracy to what MEGA achieves. On associative recall, we find that even a
  simplified version of TCNCA, without excessive multiplicative and additive
  interactions, remains superior or competitive to MEGA on a range of sequence
  lengths and vocabulary sizes.
paper_authors._id: 6577daa276b6de7978f2e9ae
paper_authors.name: Aleksandar Terzic
paper_authors._id: 6577daa276b6de7978f2e9af
paper_authors.name: Michael Hersche
paper_authors._id: 6577daa276b6de7978f2e9b0
paper_authors.name: Geethan Karunaratne
paper_authors._id: 6577daa276b6de7978f2e9b1
paper_authors.name: Luca Benini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T09:09:19.664Z
paper_authors._id: 64a6a9370602161aa9090ecd
paper_authors.avatarUrl: /avatars/8488546102fefb5c5b218469e6f886c0.svg
paper_authors.fullname: Luca Benini
paper_authors.type: user
paper_authors.user: lbenini
paper_authors._id: 6577daa276b6de7978f2e9b2
paper_authors.name: Abu Sebastian
paper_authors._id: 6577daa276b6de7978f2e9b3
paper_authors.name: Abbas Rahimi
paper_title: TCNCA: Temporal Convolution Network with Chunked Attention for Scalable
    Sequence Processing
paper_summary: MEGA is a recent transformer-based architecture, which utilizes a linear
  recurrent operator whose parallel computation, based on the FFT, scales as
  O(LlogL), with L being the sequence length. We build upon their approach by
  replacing the linear recurrence with a special temporal convolutional network
  which permits larger receptive field size with shallower networks, and reduces
  the computational complexity to O(L). The resulting model is called TCNCA, a
  Temporal Convolutional Network with Chunked Attention. We evaluate TCNCA on
  EnWik8 language modeling, long-range-arena (LRA) sequence classification, as
  well as a synthetic reasoning benchmark associative recall. On EnWik8, TCNCA
  outperforms MEGA, reaching a lower loss with 1.37times/1.24times faster
  forward/backward pass during training. The dilated convolutions used in TCNCA
  are consistently and significantly faster operations than the FFT-based
  parallelized recurrence in GPUs, making them a scalable candidate for handling
  very large sequence lengths: they are up to 7.07times/2.86times faster in
  the forward/backward pass for sequences up to 131k. Further on LRA, TCNCA
  achieves, on average, 1.28times speed-up during inference with similar
  accuracy to what MEGA achieves. On associative recall, we find that even a
  simplified version of TCNCA, without excessive multiplicative and additive
  interactions, remains superior or competitive to MEGA on a range of sequence
  lengths and vocabulary sizes.
paper_ai_keywords: context transformer-based architecture linear recurrent operator FFT (Fast Fourier Transform) temporal convolutional network orthogonal TCNCA (Temporal Convolutional Network with Chunked Attention) EnWik8 language modeling long-range-arena (LRA) sequence classification associative recall dilated convolutions GPU forward/backward pass inference multiplicative and additive interactions