title: OLMoE: Open Mixture-of-Experts Language Models
summary: We introduce OLMoE, a fully open, state-of-the-art language model leveraging
  sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but
  uses only 1B per input token. We pretrain it on 5 trillion tokens and further
  adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available
  models with similar active parameters, even surpassing larger ones like
  Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE
  training, analyze routing in our model showing high specialization, and
  open-source all aspects of our work: model weights, training data, code, and
  logs.
paper_authors._id: 66d7c90fe1aee6b7145bd5a3
paper_authors.name: Niklas Muennighoff
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-04T02:49:16.535Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 66d7c90fe1aee6b7145bd5a4
paper_authors.name: Luca Soldaini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:04:06.994Z
paper_authors._id: 5f04d8c45d08220171a0ad32
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
paper_authors.fullname: Luca Soldaini
paper_authors.type: user
paper_authors.user: soldni
paper_authors._id: 66d7c90fe1aee6b7145bd5a5
paper_authors.name: Dirk Groeneveld
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:04:13.711Z
paper_authors._id: 60369745413a78f892e7339c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
paper_authors.fullname: Dirk Groeneveld
paper_authors.type: user
paper_authors.user: dirkgr
paper_authors._id: 66d7c90fe1aee6b7145bd5a6
paper_authors.name: Kyle Lo
paper_authors._id: 66d7c90fe1aee6b7145bd5a7
paper_authors.name: Jacob Morrison
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:05:12.456Z
paper_authors._id: 63f51f6cb51da4d61da72937
paper_authors.avatarUrl: /avatars/6023ecaafb28b7599f59816381d58bd0.svg
paper_authors.fullname: Jacob Morrison
paper_authors.type: user
paper_authors.user: jacobmorrison
paper_authors._id: 66d7c90fe1aee6b7145bd5a8
paper_authors.name: Sewon Min
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:05:30.534Z
paper_authors._id: 63a76d0de27a6dbd485fe863
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a76d0de27a6dbd485fe863/qJJwHOuvyQGq1o0KscOF_.jpeg
paper_authors.fullname: Sewon Min
paper_authors.type: user
paper_authors.user: sewon
paper_authors._id: 66d7c90fe1aee6b7145bd5a9
paper_authors.name: Weijia Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:05:44.148Z
paper_authors._id: 6400f2ed568dbe30c9161e47
paper_authors.avatarUrl: /avatars/c55938df5bce82b5d96e592a1ec36a8b.svg
paper_authors.fullname: Weijia Shi
paper_authors.type: user
paper_authors.user: swj0419
paper_authors._id: 66d7c90fe1aee6b7145bd5aa
paper_authors.name: Pete Walsh
paper_authors._id: 66d7c90fe1aee6b7145bd5ab
paper_authors.name: Oyvind Tafjord
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:05:54.636Z
paper_authors._id: 609b1aeff7c1303cfd1760f3
paper_authors.avatarUrl: /avatars/3cb448514885656466886d7c84e12af1.svg
paper_authors.fullname: Oyvind Tafjord
paper_authors.type: user
paper_authors.user: OyvindTafjord
paper_authors._id: 66d7c90fe1aee6b7145bd5ac
paper_authors.name: Nathan Lambert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:06:03.121Z
paper_authors._id: 628e5f90a9a3c754c1f7c88f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
paper_authors.fullname: Nathan Lambert
paper_authors.type: user
paper_authors.user: natolambert
paper_authors._id: 66d7c90fe1aee6b7145bd5ad
paper_authors.name: Yuling Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:06:11.050Z
paper_authors._id: 62a8b91c1277a49db0c5c080
paper_authors.avatarUrl: /avatars/9cf834a02b67af862659c9b9d4367a52.svg
paper_authors.fullname: Yuling Gu
paper_authors.type: user
paper_authors.user: SaveBertAndGpt
paper_authors._id: 66d7c90fe1aee6b7145bd5ae
paper_authors.name: Shane Arora
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:06:26.945Z
paper_authors._id: 65c27f58fe2f2618f82a69bf
paper_authors.avatarUrl: /avatars/53c79c0d09f36e33b8557f649d2b7261.svg
paper_authors.fullname: Shane Arora
paper_authors.type: user
paper_authors.user: shanearora
paper_authors._id: 66d7c90fe1aee6b7145bd5af
paper_authors.name: Akshita Bhagia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:06:32.853Z
paper_authors._id: 6282937a9624ab5f75c977e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
paper_authors.fullname: Akshita Bhagia
paper_authors.type: user
paper_authors.user: akshitab
paper_authors._id: 66d7c90fe1aee6b7145bd5b0
paper_authors.name: Dustin Schwenk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:06:38.370Z
paper_authors._id: 647e791b7f9ad5e44babb965
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647e791b7f9ad5e44babb965/Vjfy4c3ooDwU24N7mvRdy.jpeg
paper_authors.fullname: Dustin Schwenk
paper_authors.type: user
paper_authors.user: drschwenk
paper_authors._id: 66d7c90fe1aee6b7145bd5b1
paper_authors.name: David Wadden
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:06:43.850Z
paper_authors._id: 5ff4e2a1463be69ae4bd42bd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1609884433338-5ff4e2a1463be69ae4bd42bd.jpeg
paper_authors.fullname: David Wadden
paper_authors.type: user
paper_authors.user: dwadden
paper_authors._id: 66d7c90fe1aee6b7145bd5b2
paper_authors.name: Alexander Wettig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:06:50.080Z
paper_authors._id: 6491fd16fe1c0c0843df62c0
paper_authors.avatarUrl: /avatars/a049fe3ab75cb91fa95c33064cf1a4d9.svg
paper_authors.fullname: Alexander Wettig
paper_authors.type: user
paper_authors.user: awettig
paper_authors._id: 66d7c90fe1aee6b7145bd5b3
paper_authors.name: Binyuan Hui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:06:56.130Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 66d7c90fe1aee6b7145bd5b4
paper_authors.name: Tim Dettmers
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:07:03.191Z
paper_authors._id: 61a8f52d89ea472e600a0a10
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1638462736111-noauth.png
paper_authors.fullname: Tim Dettmers
paper_authors.type: user
paper_authors.user: timdettmers
paper_authors._id: 66d7c90fe1aee6b7145bd5b5
paper_authors.name: Douwe Kiela
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:07:09.260Z
paper_authors._id: 61dc997715b47073db1620dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1641847245435-61dc997715b47073db1620dc.jpeg
paper_authors.fullname: Douwe Kiela
paper_authors.type: user
paper_authors.user: douwekiela
paper_authors._id: 66d7c90fe1aee6b7145bd5b6
paper_authors.name: Ali Farhadi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:07:15.914Z
paper_authors._id: 6660d4c1818c5c5ca0f31266
paper_authors.avatarUrl: /avatars/1d2972894cb3b9df1900fdb162d9c364.svg
paper_authors.fullname: alifarhadi 
paper_authors.type: user
paper_authors.user: alifarhadi051
paper_authors._id: 66d7c90fe1aee6b7145bd5b7
paper_authors.name: Noah A. Smith
paper_authors._id: 66d7c90fe1aee6b7145bd5b8
paper_authors.name: Pang Wei Koh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:07:28.876Z
paper_authors._id: 641b4263abfce26bcf7b27de
paper_authors.avatarUrl: /avatars/e91b4205e4f74b0dd8c333c23203a924.svg
paper_authors.fullname: Pang Wei Koh
paper_authors.type: user
paper_authors.user: pangwei
paper_authors._id: 66d7c90fe1aee6b7145bd5b9
paper_authors.name: Amanpreet Singh
paper_authors._id: 66d7c90fe1aee6b7145bd5ba
paper_authors.name: Hannaneh Hajishirzi
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-09-04T02:42:24.466Z
paper_authors._id: 65ea4989fddceb72b1082fc0
paper_authors.avatarUrl: /avatars/0b4186eb8bf3fa00989e3f5015550d54.svg
paper_authors.fullname: Hanna Hajishirzi
paper_authors.type: user
paper_authors.user: hanna-ai2
paper_title: OLMoE: Open Mixture-of-Experts Language Models
paper_summary: We introduce OLMoE, a fully open, state-of-the-art language model leveraging
  sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but
  uses only 1B per input token. We pretrain it on 5 trillion tokens and further
  adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available
  models with similar active parameters, even surpassing larger ones like
  Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE
  training, analyze routing in our model showing high specialization, and
  open-source all aspects of our work: model weights, training data, code, and
  logs.
paper_ai_keywords: sparse active parameters sparse Mixture-of-Experts (MoE) pretrain adaptation high specialization