title: Tandem Transformers for Inference Efficient LLMs
summary: The autoregressive nature of conventional large language models (LLMs)
  inherently limits inference speed, as tokens are generated sequentially. While
  speculative and parallel decoding techniques attempt to mitigate this, they
  face limitations: either relying on less accurate smaller models for generation
  or failing to fully leverage the base LLM's representations.
    We introduce a novel architecture, Tandem transformers, to address these
  issues. This architecture uniquely combines (1) a small autoregressive model
  and (2) a large model operating in block mode (processing multiple tokens
  simultaneously). The small model's predictive accuracy is substantially
  enhanced by granting it attention to the large model's richer representations.
  On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko
  demonstrates a 3.3% improvement in next-token prediction accuracy over a
  standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter
  model with comparable downstream performance. We further incorporate the tandem
  model within the speculative decoding (SPEED) framework where the large model
  validates tokens from the small model. This ensures that the Tandem of
  PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster
  than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream
  task accuracy.
paper_authors._id: 65cc2f3e33f18247f2b46345
paper_authors.name: Aishwarya P S
paper_authors._id: 65cc2f3e33f18247f2b46346
paper_authors.name: Pranav Ajit Nair
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T12:42:48.951Z
paper_authors._id: 6412ed98b2f384688cafac84
paper_authors.avatarUrl: /avatars/35fe3a4b043a0da713689a3d138d98f9.svg
paper_authors.fullname: Pranav Ajit Nair
paper_authors.type: user
paper_authors.user: pranavn10
paper_authors._id: 65cc2f3e33f18247f2b46347
paper_authors.name: Yashas Samaga
paper_authors._id: 65cc2f3e33f18247f2b46348
paper_authors.name: Toby Boyd
paper_authors._id: 65cc2f3e33f18247f2b46349
paper_authors.name: Sanjiv Kumar
paper_authors._id: 65cc2f3e33f18247f2b4634a
paper_authors.name: Prateek Jain
paper_authors._id: 65cc2f3e33f18247f2b4634b
paper_authors.name: Praneeth Netrapalli
paper_title: Tandem Transformers for Inference Efficient LLMs
paper_summary: The autoregressive nature of conventional large language models (LLMs)
  inherently limits inference speed, as tokens are generated sequentially. While
  speculative and parallel decoding techniques attempt to mitigate this, they
  face limitations: either relying on less accurate smaller models for generation
  or failing to fully leverage the base LLM's representations.
    We introduce a novel architecture, Tandem transformers, to address these
  issues. This architecture uniquely combines (1) a small autoregressive model
  and (2) a large model operating in block mode (processing multiple tokens
  simultaneously). The small model's predictive accuracy is substantially
  enhanced by granting it attention to the large model's richer representations.
  On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko
  demonstrates a 3.3% improvement in next-token prediction accuracy over a
  standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter
  model with comparable downstream performance. We further incorporate the tandem
  model within the speculative decoding (SPEED) framework where the large model
  validates tokens from the small model. This ensures that the Tandem of
  PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster
  than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream
  task accuracy.
paper_ai_summary: Tandem transformers combine an autoregressive model with a block-mode model to enhance inference speed and accuracy in language models.
paper_ai_keywords: autoregressive models LLMs speculative decoding parallel decoding block mode attention next-token prediction accuracy PaLM2 PaLM2-Bison PaLM2-Gecko PaLM2-Otter Tandem of PaLM2-Bison and PaLM2-Gecko SPEED framework