Tandem Transformers for Inference Efficient LLMs
The autoregressive nature of conventional large language models (LLMs)
inherently limits inference speed, as tokens are generated sequentially. While
speculative and parallel decoding techniques attempt to mitigate this, they
face limitations: either relying on less accurate smaller models for generation
or failing to fully leverage the base LLM's representations.
  We introduce a novel architecture, Tandem transformers, to address these
issues. This architecture uniquely combines (1) a small autoregressive model
and (2) a large model operating in block mode (processing multiple tokens
simultaneously). The small model's predictive accuracy is substantially
enhanced by granting it attention to the large model's richer representations.
On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko
demonstrates a 3.3% improvement in next-token prediction accuracy over a
standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter
model with comparable downstream performance. We further incorporate the tandem
model within the speculative decoding (SPEED) framework where the large model
validates tokens from the small model. This ensures that the Tandem of
PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster
than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream
task accuracy.
65cc2f3e33f18247f2b46345
Aishwarya P S
65cc2f3e33f18247f2b46346
Pranav Ajit Nair
admin_assigned
2024-02-14T12:42:48.951Z
6412ed98b2f384688cafac84
/avatars/35fe3a4b043a0da713689a3d138d98f9.svg
Pranav Ajit Nair
user
pranavn10
65cc2f3e33f18247f2b46347
Yashas Samaga
65cc2f3e33f18247f2b46348
Toby Boyd
65cc2f3e33f18247f2b46349
Sanjiv Kumar
65cc2f3e33f18247f2b4634a
Prateek Jain
65cc2f3e33f18247f2b4634b
Praneeth Netrapalli
Tandem Transformers for Inference Efficient LLMs
The autoregressive nature of conventional large language models (LLMs)
inherently limits inference speed, as tokens are generated sequentially. While
speculative and parallel decoding techniques attempt to mitigate this, they
face limitations: either relying on less accurate smaller models for generation
or failing to fully leverage the base LLM's representations.
  We introduce a novel architecture, Tandem transformers, to address these
issues. This architecture uniquely combines (1) a small autoregressive model
and (2) a large model operating in block mode (processing multiple tokens
simultaneously). The small model's predictive accuracy is substantially
enhanced by granting it attention to the large model's richer representations.
On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko
demonstrates a 3.3% improvement in next-token prediction accuracy over a
standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter
model with comparable downstream performance. We further incorporate the tandem
model within the speculative decoding (SPEED) framework where the large model
validates tokens from the small model. This ensures that the Tandem of
PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster
than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream
task accuracy.
Tandem transformers combine an autoregressive model with a block-mode model to enhance inference speed and accuracy in language models.
autoregressive models LLMs speculative decoding parallel decoding block mode attention next-token prediction accuracy PaLM2 PaLM2-Bison PaLM2-Gecko PaLM2-Otter Tandem of PaLM2-Bison and PaLM2-Gecko SPEED framework