Energy-Based Transformers are Scalable Learners and Thinkers
Inference-time computation techniques, analogous to human System 2 Thinking,
have recently become popular for improving model performances. However, most
existing approaches suffer from several limitations: they are modality-specific
(e.g., working only in text), problem-specific (e.g., verifiable domains like
math and coding), or require additional supervision/training on top of
unsupervised pretraining (e.g., verifiers or verifiable rewards). In this
paper, we ask the question "Is it possible to generalize these System 2
Thinking approaches, and develop models that learn to think solely from
unsupervised learning?" Interestingly, we find the answer is yes, by learning
to explicitly verify the compatibility between inputs and
candidate-predictions, and then re-framing prediction problems as optimization
with respect to this verifier. Specifically, we train Energy-Based Transformers
(EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy
value to every input and candidate-prediction pair, enabling predictions
through gradient descent-based energy minimization until convergence. Across
both discrete (text) and continuous (visual) modalities, we find EBTs scale
faster than the dominant Transformer++ approach during training, achieving an
up to 35% higher scaling rate with respect to data, batch size, parameters,
FLOPs, and depth. During inference, EBTs improve performance with System 2
Thinking by 29% more than the Transformer++ on language tasks, and EBTs
outperform Diffusion Transformers on image denoising while using fewer forward
passes. Further, we find that EBTs achieve better results than existing models
on most downstream tasks given the same or worse pretraining performance,
suggesting that EBTs generalize better than existing approaches. Consequently,
EBTs are a promising new paradigm for scaling both the learning and thinking
capabilities of models.
68676e4a9db35afc9c304d72
Alexi Gladstone
claimed_verified
2025-07-05T07:53:10.646Z
63fbbff41b4b1bd4e7042fd1
/avatars/d186d0859046b35723fe1156d8f68917.svg
Alexi Gladstone
user
alexiglad
68676e4a9db35afc9c304d73
Ganesh Nanduru
68676e4a9db35afc9c304d74
Md Mofijul Islam
68676e4a9db35afc9c304d75
Peixuan Han
68676e4a9db35afc9c304d76
Hyeonjeong Ha
68676e4a9db35afc9c304d77
Aman Chadha
claimed_verified
2025-07-04T07:48:46.365Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
68676e4a9db35afc9c304d78
Yilun Du
68676e4a9db35afc9c304d79
Heng Ji
68676e4a9db35afc9c304d7a
Jundong Li
68676e4a9db35afc9c304d7b
Tariq Iqbal
Energy-Based Transformers are Scalable Learners and Thinkers
Inference-time computation techniques, analogous to human System 2 Thinking,
have recently become popular for improving model performances. However, most
existing approaches suffer from several limitations: they are modality-specific
(e.g., working only in text), problem-specific (e.g., verifiable domains like
math and coding), or require additional supervision/training on top of
unsupervised pretraining (e.g., verifiers or verifiable rewards). In this
paper, we ask the question "Is it possible to generalize these System 2
Thinking approaches, and develop models that learn to think solely from
unsupervised learning?" Interestingly, we find the answer is yes, by learning
to explicitly verify the compatibility between inputs and
candidate-predictions, and then re-framing prediction problems as optimization
with respect to this verifier. Specifically, we train Energy-Based Transformers
(EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy
value to every input and candidate-prediction pair, enabling predictions
through gradient descent-based energy minimization until convergence. Across
both discrete (text) and continuous (visual) modalities, we find EBTs scale
faster than the dominant Transformer++ approach during training, achieving an
up to 35% higher scaling rate with respect to data, batch size, parameters,
FLOPs, and depth. During inference, EBTs improve performance with System 2
Thinking by 29% more than the Transformer++ on language tasks, and EBTs
outperform Diffusion Transformers on image denoising while using fewer forward
passes. Further, we find that EBTs achieve better results than existing models
on most downstream tasks given the same or worse pretraining performance,
suggesting that EBTs generalize better than existing approaches. Consequently,
EBTs are a promising new paradigm for scaling both the learning and thinking
capabilities of models.
Energy-Based Transformers (EBTs) improve model performance and scalability across modalities by learning to verify predictions through unsupervised learning and energy minimization.
Energy-Based Transformers Energy-Based Models gradient descent-based energy minimization System 2 Thinking Transformer++ Diffusion Transformers image denoising downstream tasks pretraining performance