title: MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model
    Series
summary: Large Language Models (LLMs) have made great strides in recent years to
  achieve unprecedented performance across different tasks. However, due to
  commercial interest, the most competitive models like GPT, Gemini, and Claude
  have been gated behind proprietary interfaces without disclosing the training
  details. Recently, many institutions have open-sourced several strong LLMs like
  LLaMA-3, comparable to existing closed-source LLMs. However, only the model's
  weights are provided with most details (e.g., intermediate checkpoints,
  pre-training corpus, and training code, etc.) being undisclosed. To improve the
  transparency of LLMs, the research community has formed to open-source truly
  open LLMs (e.g., Pythia, Amber, OLMo), where more details (e.g., pre-training
  corpus and training code) are being provided. These models have greatly
  advanced the scientific study of these large models including their strengths,
  weaknesses, biases and risks. However, we observe that the existing truly open
  LLMs on reasoning, knowledge, and coding tasks are still inferior to existing
  state-of-the-art LLMs with similar model sizes. To this end, we open-source
  MAP-Neo, a highly capable and transparent bilingual language model with 7B
  parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is the
  first fully open-sourced bilingual LLM with comparable performance compared to
  existing state-of-the-art LLMs. Moreover, we open-source all details to
  reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning
  pipeline, checkpoints, and well-optimized training/evaluation framework are
  provided. Finally, we hope our MAP-Neo will enhance and strengthen the open
  research community and inspire more innovations and creativities to facilitate
  the further improvements of LLMs.
paper_authors._id: 6657e84f1436864bccbdcc40
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:25:40.803Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6657e84f1436864bccbdcc41
paper_authors.name: Scott Qu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:25:42.762Z
paper_authors._id: 646b4c9fdf2609a541c0866e
paper_authors.avatarUrl: /avatars/7ec6c709017dcf32d4ac49d1e3820328.svg
paper_authors.fullname: Qu
paper_authors.type: user
paper_authors.user: ScottQu
paper_authors._id: 6657e84f1436864bccbdcc42
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:16:49.808Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 6657e84f1436864bccbdcc43
paper_authors.name: Chenchen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-02T18:47:30.655Z
paper_authors._id: 64b74b906ab5d14ca7f289cd
paper_authors.avatarUrl: /avatars/b131b7c4ce5216708ca4a678f35ead0a.svg
paper_authors.fullname: xxzcc
paper_authors.type: user
paper_authors.user: xxzcc
paper_authors._id: 6657e84f1436864bccbdcc44
paper_authors.name: Chenghua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:23:35.232Z
paper_authors._id: 6442ba123610a28a4ad5382f
paper_authors.avatarUrl: /avatars/97f2d3d2c10058ea41d7bf9087e1f619.svg
paper_authors.fullname: Chenghua Lin
paper_authors.type: user
paper_authors.user: chenghualin
paper_authors._id: 6657e84f1436864bccbdcc45
paper_authors.name: Chou Leuang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T06:47:55.181Z
paper_authors._id: 62a80fe3ac97233f1625235a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a80fe3ac97233f1625235a/_rGtpqdY7OEBz3pyqb6fE.jpeg
paper_authors.fullname: Zhouliang Yu
paper_authors.type: user
paper_authors.user: zhouliang
paper_authors._id: 6657e84f1436864bccbdcc46
paper_authors.name: Danny Pan
paper_authors._id: 6657e84f1436864bccbdcc47
paper_authors.name: Esther Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:24:11.283Z
paper_authors._id: 651fa34245d808ba81132cdb
paper_authors.avatarUrl: /avatars/444d2cb7e4e4b3c32555886dc23fc512.svg
paper_authors.fullname: Esther
paper_authors.type: user
paper_authors.user: EstherCheng
paper_authors._id: 6657e84f1436864bccbdcc48
paper_authors.name: Jie Liu
paper_authors._id: 6657e84f1436864bccbdcc49
paper_authors.name: Qunshu Lin
paper_authors._id: 6657e84f1436864bccbdcc4a
paper_authors.name: Raven Yuan
paper_authors._id: 6657e84f1436864bccbdcc4b
paper_authors.name: Tuney Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:25:38.830Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 6657e84f1436864bccbdcc4c
paper_authors.name: Wei Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:02:51.659Z
paper_authors._id: 65164444bc0631719873af81
paper_authors.avatarUrl: /avatars/dab8b90db8bbd00806268fe276e3ea36.svg
paper_authors.fullname: Wei Pang
paper_authors.type: user
paper_authors.user: weipang142857
paper_authors._id: 6657e84f1436864bccbdcc4d
paper_authors.name: Xinrun Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:25:45.759Z
paper_authors._id: 654907a4a1faff97850c4eff
paper_authors.avatarUrl: /avatars/458c90151614bc7f116943b6e67d6b8a.svg
paper_authors.fullname: du
paper_authors.type: user
paper_authors.user: dododododo
paper_authors._id: 6657e84f1436864bccbdcc4e
paper_authors.name: Yiming Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:25:32.380Z
paper_authors._id: 654fed4d7490049d621d0cd3
paper_authors.avatarUrl: /avatars/6bc1553d1571ab4e9ab80c58eeb9820c.svg
paper_authors.fullname: Yiming Liang
paper_authors.type: user
paper_authors.user: Yiming1234
paper_authors._id: 6657e84f1436864bccbdcc4f
paper_authors.name: Yinghao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:25:40.489Z
paper_authors._id: 6410665d5364a661bee22524
paper_authors.avatarUrl: /avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
paper_authors.fullname: Yinghao Ma
paper_authors.type: user
paper_authors.user: nicolaus625
paper_authors._id: 6657e84f1436864bccbdcc50
paper_authors.name: Yizhi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:25:48.259Z
paper_authors._id: 6382252f54421460665ec501
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
paper_authors.fullname: Yizhi Li
paper_authors.type: user
paper_authors.user: yizhilll
paper_authors._id: 6657e84f1436864bccbdcc51
paper_authors.name: Ziyang Ma
paper_authors._id: 6657e84f1436864bccbdcc52
paper_authors.name: Bill Lin
paper_authors._id: 6657e84f1436864bccbdcc53
paper_authors.name: Emmanouil Benetos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-05T10:24:08.573Z
paper_authors._id: 66b0a0839797b94f001ed874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
paper_authors.fullname: Emmanouil Benetos
paper_authors.type: user
paper_authors.user: emmanouilb
paper_authors._id: 6657e84f1436864bccbdcc54
paper_authors.name: Huan Yang
paper_authors._id: 6657e84f1436864bccbdcc55
paper_authors.name: Junting Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:26:53.213Z
paper_authors._id: 65a374a59acab1998092a9bc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a374a59acab1998092a9bc/M3s_7bSf9G-6b9nLg7N3Z.jpeg
paper_authors.fullname: Antonio
paper_authors.type: user
paper_authors.user: JuntingZhou
paper_authors._id: 6657e84f1436864bccbdcc56
paper_authors.name: Kaijing Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:27:00.457Z
paper_authors._id: 65eb65722fbf6807134a636c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65eb65722fbf6807134a636c/8YgrynSM-WExVlu2mwRBY.jpeg
paper_authors.fullname: Kaijing Ma
paper_authors.type: user
paper_authors.user: mkj69
paper_authors._id: 6657e84f1436864bccbdcc57
paper_authors.name: Minghao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T11:47:36.314Z
paper_authors._id: 6417d9ea8f689506e7148417
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
paper_authors.fullname: minghao
paper_authors.type: user
paper_authors.user: Liam-Liu
paper_authors._id: 6657e84f1436864bccbdcc58
paper_authors.name: Morry Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:27:37.870Z
paper_authors._id: 65bb14f139c4e7087640a91c
paper_authors.avatarUrl: /avatars/dbf75dd161d22b4511e9fccff6afc515.svg
paper_authors.fullname: Morry Niu
paper_authors.type: user
paper_authors.user: bl1ndbot
paper_authors._id: 6657e84f1436864bccbdcc59
paper_authors.name: Noah Wang
paper_authors._id: 6657e84f1436864bccbdcc5a
paper_authors.name: Quehry Que
paper_authors._id: 6657e84f1436864bccbdcc5b
paper_authors.name: Ruibo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:28:16.504Z
paper_authors._id: 5e91f1554957053f606489ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e91f1554957053f606489ff/eJty7TksRLwuta0deZTmt.jpeg
paper_authors.fullname: Ruibo Liu
paper_authors.type: user
paper_authors.user: agi-css
paper_authors._id: 6657e84f1436864bccbdcc5c
paper_authors.name: Sine Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:29:57.165Z
paper_authors._id: 618e3adaa834a6636ab624c2
paper_authors.avatarUrl: /avatars/107110d8384e5c2e2f5021bdf2c5a79c.svg
paper_authors.fullname: Siyao Liu
paper_authors.type: user
paper_authors.user: sine
paper_authors._id: 6657e84f1436864bccbdcc5d
paper_authors.name: Shawn Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:30:04.610Z
paper_authors._id: 61b8402cae47fface62479ae
paper_authors.avatarUrl: /avatars/deaef201a6e0844c6805bc8c124c5308.svg
paper_authors.fullname: Shawn Guo
paper_authors.type: user
paper_authors.user: shawnGuo
paper_authors._id: 6657e84f1436864bccbdcc5e
paper_authors.name: Soren Gao
paper_authors._id: 6657e84f1436864bccbdcc5f
paper_authors.name: Wangchunshu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:30:22.948Z
paper_authors._id: 628c8598ef14f971b698107f
paper_authors.avatarUrl: /avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Wangchunshu
paper_authors._id: 6657e84f1436864bccbdcc60
paper_authors.name: Xinyue Zhang
paper_authors._id: 6657e84f1436864bccbdcc61
paper_authors.name: Yizhi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:31:02.006Z
paper_authors._id: 634182c506a304bdc43037c8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665237671855-noauth.png
paper_authors.fullname: YIZHI ZHOU
paper_authors.type: user
paper_authors.user: IanZhou11
paper_authors._id: 6657e84f1436864bccbdcc62
paper_authors.name: Yubo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T08:41:39.648Z
paper_authors._id: 636a35eff8d9af4aea181608
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
paper_authors.fullname: yubo
paper_authors.type: user
paper_authors.user: ubowang
paper_authors._id: 6657e84f1436864bccbdcc63
paper_authors.name: Yuelin Bai
paper_authors._id: 6657e84f1436864bccbdcc64
paper_authors.name: Yuhan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:32:02.704Z
paper_authors._id: 6405e13aa577649430bd3f47
paper_authors.avatarUrl: /avatars/de63f5f5e6a0917d63a34395323063cd.svg
paper_authors.fullname: Yuhan Zhang
paper_authors.type: user
paper_authors.user: hemomo
paper_authors._id: 6657e84f1436864bccbdcc65
paper_authors.name: Yuxiang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:12:37.879Z
paper_authors._id: 6239386535384c2bcccd2a4f
paper_authors.avatarUrl: /avatars/c62cefc6235210ca12b46beb62751d8b.svg
paper_authors.fullname: Yuxiang Zhang
paper_authors.type: user
paper_authors.user: Joelzhang
paper_authors._id: 6657e84f1436864bccbdcc66
paper_authors.name: Zenith Wang
paper_authors._id: 6657e84f1436864bccbdcc67
paper_authors.name: Zhenzhu Yang
paper_authors._id: 6657e84f1436864bccbdcc68
paper_authors.name: Zijian Zhao
paper_authors._id: 6657e84f1436864bccbdcc69
paper_authors.name: Jiajun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:34:09.599Z
paper_authors._id: 63f108072f7c0152e86fa2ea
paper_authors.avatarUrl: /avatars/992ec3f9642df1688c5b20b3f6c938dd.svg
paper_authors.fullname: Jiajun Zhang
paper_authors.type: user
paper_authors.user: jjzhang
paper_authors._id: 6657e84f1436864bccbdcc6a
paper_authors.name: Wanli Ouyang
paper_authors._id: 6657e84f1436864bccbdcc6b
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:30:43.392Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 6657e84f1436864bccbdcc6c
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:32:41.678Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model
    Series
paper_summary: Large Language Models (LLMs) have made great strides in recent years to
  achieve unprecedented performance across different tasks. However, due to
  commercial interest, the most competitive models like GPT, Gemini, and Claude
  have been gated behind proprietary interfaces without disclosing the training
  details. Recently, many institutions have open-sourced several strong LLMs like
  LLaMA-3, comparable to existing closed-source LLMs. However, only the model's
  weights are provided with most details (e.g., intermediate checkpoints,
  pre-training corpus, and training code, etc.) being undisclosed. To improve the
  transparency of LLMs, the research community has formed to open-source truly
  open LLMs (e.g., Pythia, Amber, OLMo), where more details (e.g., pre-training
  corpus and training code) are being provided. These models have greatly
  advanced the scientific study of these large models including their strengths,
  weaknesses, biases and risks. However, we observe that the existing truly open
  LLMs on reasoning, knowledge, and coding tasks are still inferior to existing
  state-of-the-art LLMs with similar model sizes. To this end, we open-source
  MAP-Neo, a highly capable and transparent bilingual language model with 7B
  parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is the
  first fully open-sourced bilingual LLM with comparable performance compared to
  existing state-of-the-art LLMs. Moreover, we open-source all details to
  reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning
  pipeline, checkpoints, and well-optimized training/evaluation framework are
  provided. Finally, we hope our MAP-Neo will enhance and strengthen the open
  research community and inspire more innovations and creativities to facilitate
  the further improvements of LLMs.
paper_ai_summary: MAP-Neo, a fully open-sourced bilingual LLM with 7B parameters, achieves performance comparable to existing state-of-the-art LLMs, enhancing transparency and fostering innovation in the open research community.
paper_ai_keywords: Large Language Models LLMs GPT Gemini Claude LLaMA-3 Pythia Amber OLMo MAP-Neo bilingual language model pre-training corpus data cleaning pipeline checkpoints training/evaluation framework