title: Block-State Transformer
summary: State space models (SSMs) have shown impressive results on tasks that require
  modeling long-range dependencies and efficiently scale to long sequences owing
  to their subquadratic runtime complexity. Originally designed for continuous
  signals, SSMs have shown superior performance on a plethora of tasks, in vision
  and audio; however, SSMs still lag Transformer performance in Language Modeling
  tasks. In this work, we propose a hybrid layer named Block-State Transformer
  (BST), that internally combines an SSM sublayer for long-range
  contextualization, and a Block Transformer sublayer for short-term
  representation of sequences. We study three different, and completely
  parallelizable, variants that integrate SSMs and block-wise attention. We show
  that our model outperforms similar Transformer-based architectures on language
  modeling perplexity and generalizes to longer sequences. In addition, the
  Block-State Transformer demonstrates more than tenfold increase in speed at the
  layer level compared to the Block-Recurrent Transformer when model
  parallelization is employed.
paper_authors._id: 648fb819cda0395a6f332943
paper_authors.name: Mahan Fathi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-19T11:44:42.951Z
paper_authors._id: 64903c2d9b1aaa0738286323
paper_authors.avatarUrl: /avatars/9c4d0734b6f779d0f138951998f1d7b5.svg
paper_authors.fullname: Mahan Fathi
paper_authors.type: user
paper_authors.user: mahanfathi
paper_authors._id: 648fb819cda0395a6f332944
paper_authors.name: Jonathan Pilault
paper_authors._id: 648fb819cda0395a6f332945
paper_authors.name: Pierre-Luc Bacon
paper_authors._id: 648fb819cda0395a6f332946
paper_authors.name: Christopher Pal
paper_authors._id: 648fb819cda0395a6f332947
paper_authors.name: Orhan Firat
paper_authors._id: 648fb819cda0395a6f332948
paper_authors.name: Ross Goroshin
paper_title: Block-State Transformer
paper_summary: State space models (SSMs) have shown impressive results on tasks that require
  modeling long-range dependencies and efficiently scale to long sequences owing
  to their subquadratic runtime complexity. Originally designed for continuous
  signals, SSMs have shown superior performance on a plethora of tasks, in vision
  and audio; however, SSMs still lag Transformer performance in Language Modeling
  tasks. In this work, we propose a hybrid layer named Block-State Transformer
  (BST), that internally combines an SSM sublayer for long-range
  contextualization, and a Block Transformer sublayer for short-term
  representation of sequences. We study three different, and completely
  parallelizable, variants that integrate SSMs and block-wise attention. We show
  that our model outperforms similar Transformer-based architectures on language
  modeling perplexity and generalizes to longer sequences. In addition, the
  Block-State Transformer demonstrates more than tenfold increase in speed at the
  layer level compared to the Block-Recurrent Transformer when model
  parallelization is employed.
paper_ai_keywords: state space models (SSMs) long-range dependencies pixel space latent space Subquadratic runtime complexity continuous signals Block-State Transformer (BST) SSM sublayer Block Transformer sublayer long-range contextualization block-wise attention language modeling perplexity model parallelization