MixCE: Training Autoregressive Language Models by Mixing Forward and
  Reverse Cross-Entropies
Autoregressive language models are trained by minimizing the cross-entropy of
the model distribution Q relative to the data distribution P -- that is,
minimizing the forward cross-entropy, which is equivalent to maximum likelihood
estimation (MLE). We have observed that models trained in this way may
"over-generalize", in the sense that they produce non-human-like text.
Moreover, we believe that reverse cross-entropy, i.e., the cross-entropy of P
relative to Q, is a better reflection of how a human would evaluate text
generated by a model. Hence, we propose learning with MixCE, an objective that
mixes the forward and reverse cross-entropies. We evaluate models trained with
this objective on synthetic data settings (where P is known) and real data, and
show that the resulting models yield better generated text without complex
decoding strategies. Our code and models are publicly available at
https://github.com/bloomberg/mixce-acl2023
64741d21485a7c8e1bd57722
Shiyue Zhang
claimed_verified
2025-04-20T15:06:23.728Z
613a776517297f3c0bfd7b41
https://cdn-avatars.huggingface.co/v1/production/uploads/1632426687077-613a776517297f3c0bfd7b41.jpeg
Shiyue Zhang
user
shiyue
64741d21485a7c8e1bd57723
Shijie Wu
claimed_verified
2024-07-08T08:52:04.019Z
6137ad94501f80a6f6e1eac9
https://cdn-avatars.huggingface.co/v1/production/uploads/1631038777636-noauth.jpeg
Shijie Wu
user
shijie-wu
64741d21485a7c8e1bd57724
Ozan Irsoy
64741d21485a7c8e1bd57725
Steven Lu
admin_assigned
2023-05-30T09:57:06.435Z
646e4e8ecbb7bb99651180db
/avatars/0a53e18f81a6d517487477d8874aaa83.svg
Steven Lu
user
slu126
64741d21485a7c8e1bd57726
Mohit Bansal
64741d21485a7c8e1bd57727
Mark Dredze
64741d21485a7c8e1bd57728
David Rosenberg
MixCE: Training Autoregressive Language Models by Mixing Forward and
  Reverse Cross-Entropies
Autoregressive language models are trained by minimizing the cross-entropy of
the model distribution Q relative to the data distribution P -- that is,
minimizing the forward cross-entropy, which is equivalent to maximum likelihood
estimation (MLE). We have observed that models trained in this way may
"over-generalize", in the sense that they produce non-human-like text.
Moreover, we believe that reverse cross-entropy, i.e., the cross-entropy of P
relative to Q, is a better reflection of how a human would evaluate text
generated by a model. Hence, we propose learning with MixCE, an objective that
mixes the forward and reverse cross-entropies. We evaluate models trained with
this objective on synthetic data settings (where P is known) and real data, and
show that the resulting models yield better generated text without complex
decoding strategies. Our code and models are publicly available at
https://github.com/bloomberg/mixce-acl2023
Training autoregressive language models with a mixed forward and reverse cross-entropy objective improves text quality over traditional methods.
autoregressive language models cross-entropy model distribution data distribution forward cross-entropy maximum likelihood estimation reverse cross-entropy MixCE synthetic data real data