Seq vs Seq: An Open Suite of Paired Encoders and Decoders
The large language model (LLM) community focuses almost exclusively on
decoder-only language models, since they are easier to use for text generation.
However, a large subset of the community still uses encoder-only models for
tasks such as classification or retrieval. Previous work has attempted to
compare these architectures, but is forced to make comparisons with models that
have different numbers of parameters, training techniques, and datasets. We
introduce the SOTA open-data Ettin suite of models: paired encoder-only and
decoder-only models ranging from 17 million parameters to 1 billion, trained on
up to 2 trillion tokens. Using the same recipe for both encoder-only and
decoder-only models produces SOTA recipes in both categories for their
respective sizes, beating ModernBERT as an encoder and Llama 3.2 and SmolLM2 as
decoders. Like previous work, we find that encoder-only models excel at
classification and retrieval tasks while decoders excel at generative tasks.
However, we show that adapting a decoder model to encoder tasks (and vice
versa) through continued training is subpar compared to using only the reverse
objective (i.e. a 400M encoder outperforms a 1B decoder on MNLI, and vice versa
for generative tasks). We open-source all artifacts of this study including
training data, training order segmented by checkpoint, and 200+ checkpoints to
allow future work to analyze or extend all aspects of training.
6877521f257d4f043537084f
Orion Weller
6877521f257d4f0435370850
Kathryn Ricci
6877521f257d4f0435370851
Marc Marone
claimed_verified
2025-07-17T18:43:19.468Z
63e410e88083f19a218be964
/avatars/ce271d1686f6e5ee1f5b2d429cb60de6.svg
Marc Marone
user
mmarone
6877521f257d4f0435370852
Antoine Chaffin
claimed_verified
2025-07-16T15:16:06.045Z
609bbe2f4932693ca2009d6a
https://cdn-avatars.huggingface.co/v1/production/uploads/1620819560688-609bbe2f4932693ca2009d6a.jpeg
Antoine Chaffin
user
NohTow
6877521f257d4f0435370853
Dawn Lawrie
6877521f257d4f0435370854
Benjamin Van Durme
Seq vs Seq: An Open Suite of Paired Encoders and Decoders
The large language model (LLM) community focuses almost exclusively on
decoder-only language models, since they are easier to use for text generation.
However, a large subset of the community still uses encoder-only models for
tasks such as classification or retrieval. Previous work has attempted to
compare these architectures, but is forced to make comparisons with models that
have different numbers of parameters, training techniques, and datasets. We
introduce the SOTA open-data Ettin suite of models: paired encoder-only and
decoder-only models ranging from 17 million parameters to 1 billion, trained on
up to 2 trillion tokens. Using the same recipe for both encoder-only and
decoder-only models produces SOTA recipes in both categories for their
respective sizes, beating ModernBERT as an encoder and Llama 3.2 and SmolLM2 as
decoders. Like previous work, we find that encoder-only models excel at
classification and retrieval tasks while decoders excel at generative tasks.
However, we show that adapting a decoder model to encoder tasks (and vice
versa) through continued training is subpar compared to using only the reverse
objective (i.e. a 400M encoder outperforms a 1B decoder on MNLI, and vice versa
for generative tasks). We open-source all artifacts of this study including
training data, training order segmented by checkpoint, and 200+ checkpoints to
allow future work to analyze or extend all aspects of training.
The Ettin suite of models demonstrates that encoder-only and decoder-only architectures perform optimally in their respective tasks, with encoder-only models excelling at classification and retrieval, and decoder-only models at generation, and that adapting models to different tasks through continued training is less effective.
large language model decoder-only language models encoder-only models classification retrieval SOTA Ettin suite parameters tokens ModernBERT Llama 3.2 SmolLM2 MNLI generative tasks