title: OpenELM: An Efficient Language Model Family with Open-source Training
    and Inference Framework
summary: The reproducibility and transparency of large language models are crucial for
  advancing open research, ensuring the trustworthiness of results, and enabling
  investigations into data and model biases, as well as potential risks. To this
  end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a
  layer-wise scaling strategy to efficiently allocate parameters within each
  layer of the transformer model, leading to enhanced accuracy. For example, with
  a parameter budget of approximately one billion parameters, OpenELM exhibits a
  2.36% improvement in accuracy compared to OLMo while requiring 2times fewer
  pre-training tokens.
    Diverging from prior practices that only provide model weights and inference
  code, and pre-train on private datasets, our release includes the complete
  framework for training and evaluation of the language model on publicly
  available datasets, including training logs, multiple checkpoints, and
  pre-training configurations. We also release code to convert models to MLX
  library for inference and fine-tuning on Apple devices. This comprehensive
  release aims to empower and strengthen the open research community, paving the
  way for future open research endeavors.
    Our source code along with pre-trained model weights and training recipes is
  available at https://github.com/apple/corenet. Additionally, \model
  models can be found on HuggingFace at:
  https://huggingface.co/apple/OpenELM.
paper_authors._id: 662858d84a0dc08679e8f359
paper_authors.name: Sachin Mehta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:15:46.478Z
paper_authors._id: 64f792652003abb61bf5c1b8
paper_authors.avatarUrl: /avatars/9721810de3519ece50b059c154baa9ca.svg
paper_authors.fullname: Sachin Mehta
paper_authors.type: user
paper_authors.user: sacmehta
paper_authors._id: 662858d84a0dc08679e8f35a
paper_authors.name: Mohammad Hossein Sekhavat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:15:53.385Z
paper_authors._id: 65ca91986938a6a81fd80d02
paper_authors.avatarUrl: /avatars/efac7945ac1df4e8949e77465ef0dcbd.svg
paper_authors.fullname: Mohammad Hossein Sekhavat
paper_authors.type: user
paper_authors.user: msekhavat
paper_authors._id: 662858d84a0dc08679e8f35b
paper_authors.name: Qingqing Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T11:11:23.174Z
paper_authors._id: 5f07d37219cb630495b81306
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1595440980042-5f07d37219cb630495b81306.jpeg
paper_authors.fullname: Qingqing Cao
paper_authors.type: user
paper_authors.user: csarron
paper_authors._id: 662858d84a0dc08679e8f35c
paper_authors.name: Maxwell Horton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T10:39:58.055Z
paper_authors._id: 647a5afc732a0cd866eb512f
paper_authors.avatarUrl: /avatars/fbd0ba2a7ba70414ac903e583ce62933.svg
paper_authors.fullname: Maxwell Horton
paper_authors.type: user
paper_authors.user: mchorton
paper_authors._id: 662858d84a0dc08679e8f35d
paper_authors.name: Yanzi Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:15:59.745Z
paper_authors._id: 6627307fa39494ebb3a39194
paper_authors.avatarUrl: /avatars/99290642fd6d4a6162f52cfe5d46c648.svg
paper_authors.fullname: Yanzi Jin
paper_authors.type: user
paper_authors.user: yjin25
paper_authors._id: 662858d84a0dc08679e8f35e
paper_authors.name: Chenfan Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:30:02.341Z
paper_authors._id: 658895a1c4b20046639a7e59
paper_authors.avatarUrl: /avatars/f80c3f58efa5d22464590bee7a6041a4.svg
paper_authors.fullname: Chenfan Sun
paper_authors.type: user
paper_authors.user: erpang007
paper_authors._id: 662858d84a0dc08679e8f35f
paper_authors.name: Iman Mirzadeh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:16:15.272Z
paper_authors._id: 66280a4c7c988f33b795f90f
paper_authors.avatarUrl: /avatars/7a618b39c4ed20f5490ea64fbb67b5ac.svg
paper_authors.fullname: Iman Mirzadeh
paper_authors.type: user
paper_authors.user: imirzadeh-apple
paper_authors._id: 662858d84a0dc08679e8f360
paper_authors.name: Mahyar Najibi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:16:33.449Z
paper_authors._id: 661d992d69a65c0fb0717013
paper_authors.avatarUrl: /avatars/9d1f20baff742de90a50232db54d2407.svg
paper_authors.fullname: Mahyar Najibi
paper_authors.type: user
paper_authors.user: mahyar-najibi
paper_authors._id: 662858d84a0dc08679e8f361
paper_authors.name: Dmitry Belenko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T11:11:25.057Z
paper_authors._id: 63185e1a70b013d6609416ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63185e1a70b013d6609416ba/LOB6bJ71wwgamgN3MCuFb.jpeg
paper_authors.fullname: Dmitry Belenko
paper_authors.type: user
paper_authors.user: depthwise
paper_authors._id: 662858d84a0dc08679e8f362
paper_authors.name: Peter Zatloukal
paper_authors._id: 662858d84a0dc08679e8f363
paper_authors.name: Mohammad Rastegari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:16:48.273Z
paper_authors._id: 65d4c7734c2163d05eab4110
paper_authors.avatarUrl: /avatars/b116a9aca7b0a2a4836a29cb4f069d42.svg
paper_authors.fullname: Mohammad Rastegari
paper_authors.type: user
paper_authors.user: mrastegari
paper_title: OpenELM: An Efficient Language Model Family with Open-source Training
    and Inference Framework
paper_summary: The reproducibility and transparency of large language models are crucial for
  advancing open research, ensuring the trustworthiness of results, and enabling
  investigations into data and model biases, as well as potential risks. To this
  end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a
  layer-wise scaling strategy to efficiently allocate parameters within each
  layer of the transformer model, leading to enhanced accuracy. For example, with
  a parameter budget of approximately one billion parameters, OpenELM exhibits a
  2.36% improvement in accuracy compared to OLMo while requiring 2times fewer
  pre-training tokens.
    Diverging from prior practices that only provide model weights and inference
  code, and pre-train on private datasets, our release includes the complete
  framework for training and evaluation of the language model on publicly
  available datasets, including training logs, multiple checkpoints, and
  pre-training configurations. We also release code to convert models to MLX
  library for inference and fine-tuning on Apple devices. This comprehensive
  release aims to empower and strengthen the open research community, paving the
  way for future open research endeavors.
    Our source code along with pre-trained model weights and training recipes is
  available at https://github.com/apple/corenet. Additionally, \model
  models can be found on HuggingFace at:
  https://huggingface.co/apple/OpenELM.
paper_ai_summary: OpenELM, an open language model, uses layer-wise scaling to enhance accuracy and is released with comprehensive training and evaluation frameworks, enabling transparent and reproducible research.
paper_ai_keywords: layer-wise scaling transformer model OpenELM OLMo pre-training tokens training logs checkpoints pre-training configurations MLX library