title: GPT or BERT: why not both?
summary: We present a simple way to merge masked language modeling with causal
  language modeling. This hybrid training objective results in a model that
  combines the strengths of both modeling paradigms within a single transformer
  stack: GPT-BERT can be transparently used like any standard causal or masked
  language model. We test the pretraining process that enables this flexible
  behavior on the BabyLM Challenge 2024. The results show that the hybrid
  pretraining outperforms masked-only or causal-only models. We openly release
  the models, training corpora and code.
paper_authors._id: 6728d4bba53a076e4bafac1c
paper_authors.name: Lucas Georges Gabriel Charpentier
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:44:39.123Z
paper_authors._id: 64c85db9b18936750d4cb690
paper_authors.avatarUrl: /avatars/1f1f6df531300d6396f4d117cc764326.svg
paper_authors.fullname: Lucas Charpentier
paper_authors.type: user
paper_authors.user: lgcharpe
paper_authors._id: 6728d4bba53a076e4bafac1d
paper_authors.name: David Samuel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T07:53:30.031Z
paper_authors._id: 63557132525beaee688cb548
paper_authors.avatarUrl: /avatars/c67c0b1d9ea0f8a42c5c8b8cf8c20597.svg
paper_authors.fullname: David Samuel
paper_authors.type: user
paper_authors.user: davda54
paper_title: GPT or BERT: why not both?
paper_summary: We present a simple way to merge masked language modeling with causal
  language modeling. This hybrid training objective results in a model that
  combines the strengths of both modeling paradigms within a single transformer
  stack: GPT-BERT can be transparently used like any standard causal or masked
  language model. We test the pretraining process that enables this flexible
  behavior on the BabyLM Challenge 2024. The results show that the hybrid
  pretraining outperforms masked-only or causal-only models. We openly release
  the models, training corpora and code.
paper_ai_keywords: masked language modeling causal language modeling transformer stack GPT-BERT BabyLM Challenge 2024