title: XGen-7B Technical Report
summary: Large Language Models (LLMs) have become ubiquitous across various domains,
  transforming the way we interact with information and conduct research.
  However, most high-performing LLMs remain confined behind proprietary walls,
  hindering scientific progress. Most open-source LLMs, on the other hand, are
  limited in their ability to support longer sequence lengths, which is a key
  requirement for many tasks that require inference over an input context. To
  address this, we have trained XGen, a series of 7B parameter models on up to 8K
  sequence length for up to 1.5T tokens. We have also finetuned the XGen models
  on public-domain instructional data, creating their instruction-tuned
  counterparts (XGen-Inst). We open-source our models for both research
  advancements and commercial applications. Our evaluation on standard benchmarks
  shows that XGen models achieve comparable or better results when compared with
  state-of-the-art open-source LLMs. Our targeted evaluation on long sequence
  modeling tasks shows the benefits of our 8K-sequence models over 2K-sequence
  open-source LLMs.
paper_authors._id: 64fa7e88bb362cbf2fd3248b
paper_authors.name: Erik Nijkamp
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:55:46.832Z
paper_authors._id: 62472f5139c8c09b3cde0516
paper_authors.avatarUrl: /avatars/da865d0e5547fc9dc51b96153e5a8d6d.svg
paper_authors.fullname: Erik Nijkamp
paper_authors.type: user
paper_authors.user: enijkamp
paper_authors._id: 64fa7e88bb362cbf2fd3248c
paper_authors.name: Tian Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:55:55.808Z
paper_authors._id: 6401323152fb66b80d1da6c0
paper_authors.avatarUrl: /avatars/24574113bfe000250196e935e4f7cde2.svg
paper_authors.fullname: Tian Xie
paper_authors.type: user
paper_authors.user: tianxie-sf
paper_authors._id: 64fa7e88bb362cbf2fd3248d
paper_authors.name: Hiroaki Hayashi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:56:03.514Z
paper_authors._id: 619aa4e2198a95c9b962252e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648750941188-619aa4e2198a95c9b962252e.jpeg
paper_authors.fullname: Hiroaki Hayashi
paper_authors.type: user
paper_authors.user: rooa
paper_authors._id: 64fa7e88bb362cbf2fd3248e
paper_authors.name: Bo Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:56:11.852Z
paper_authors._id: 63e08acbf351dc0745749d56
paper_authors.avatarUrl: /avatars/8e2d5ce9db5bd8008ac2ad80f6025553.svg
paper_authors.fullname: Bo Pang
paper_authors.type: user
paper_authors.user: bpucla
paper_authors._id: 64fa7e88bb362cbf2fd3248f
paper_authors.name: Congying Xia
paper_authors._id: 64fa7e88bb362cbf2fd32490
paper_authors.name: Chen Xing
paper_authors._id: 64fa7e88bb362cbf2fd32491
paper_authors.name: Jesse Vig
paper_authors._id: 64fa7e88bb362cbf2fd32492
paper_authors.name: Semih Yavuz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:57:14.258Z
paper_authors._id: 64f8a68e33fd3f0fcb7810b5
paper_authors.avatarUrl: /avatars/cb5ea85e9ed513efaf24a9f36869f98f.svg
paper_authors.fullname: Semih Yavuz
paper_authors.type: user
paper_authors.user: syavuz
paper_authors._id: 64fa7e88bb362cbf2fd32493
paper_authors.name: Philippe Laban
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:57:21.555Z
paper_authors._id: 610aee2dc30263ce69e0dd9c
paper_authors.avatarUrl: /avatars/b6e5bba78425e9c28d1b75ee8a7eaac1.svg
paper_authors.fullname: Philippe Laban
paper_authors.type: user
paper_authors.user: philippelaban
paper_authors._id: 64fa7e88bb362cbf2fd32494
paper_authors.name: Ben Krause
paper_authors._id: 64fa7e88bb362cbf2fd32495
paper_authors.name: Senthil Purushwalkam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:57:33.822Z
paper_authors._id: 637e633366a51ca5d960a9b1
paper_authors.avatarUrl: /avatars/e53242e4e36875105be087c46a552e29.svg
paper_authors.fullname: Senthil Purushwalkam
paper_authors.type: user
paper_authors.user: senthilps8
paper_authors._id: 64fa7e88bb362cbf2fd32496
paper_authors.name: Tong Niu
paper_authors._id: 64fa7e88bb362cbf2fd32497
paper_authors.name: Wojciech Kryściński
paper_authors._id: 64fa7e88bb362cbf2fd32498
paper_authors.name: Lidiya Murakhovs'ka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:58:17.103Z
paper_authors._id: 60551289bf611753d3101522
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647111747809-60551289bf611753d3101522.jpeg
paper_authors.fullname: Lidiya Murakhovs'ka
paper_authors.type: user
paper_authors.user: lidiya
paper_authors._id: 64fa7e88bb362cbf2fd32499
paper_authors.name: Prafulla Kumar Choubey
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:58:43.373Z
paper_authors._id: 64e3ce5a618cd90997ed5973
paper_authors.avatarUrl: /avatars/af8709f90478c124aeef9607986f91b4.svg
paper_authors.fullname: Prafulla Kumar Choubey
paper_authors.type: user
paper_authors.user: prafullachoubey
paper_authors._id: 64fa7e88bb362cbf2fd3249a
paper_authors.name: Alex Fabbri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:58:59.370Z
paper_authors._id: 626427701ed8d81e47ab6899
paper_authors.avatarUrl: /avatars/7c89167744703771e3f163665cdb66cc.svg
paper_authors.fullname: Alexander Fabbri
paper_authors.type: user
paper_authors.user: alexfabbri
paper_authors._id: 64fa7e88bb362cbf2fd3249b
paper_authors.name: Ye Liu
paper_authors._id: 64fa7e88bb362cbf2fd3249c
paper_authors.name: Rui Meng
paper_authors._id: 64fa7e88bb362cbf2fd3249d
paper_authors.name: Lifu Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T12:00:17.653Z
paper_authors._id: 611d9648dcea95fcbbc5638e
paper_authors.avatarUrl: /avatars/e94bf70da30ed5036e2c12a38b6fcad9.svg
paper_authors.fullname: Lifu Tu
paper_authors.type: user
paper_authors.user: tulifu
paper_authors._id: 64fa7e88bb362cbf2fd3249e
paper_authors.name: Meghana Bhat
paper_authors._id: 64fa7e88bb362cbf2fd3249f
paper_authors.name: Chien-Sheng Wu
paper_authors._id: 64fa7e88bb362cbf2fd324a0
paper_authors.name: Silvio Savarese
paper_authors._id: 64fa7e88bb362cbf2fd324a1
paper_authors.name: Yingbo Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T12:01:03.197Z
paper_authors._id: 649bc93758d8b19de0c7785f
paper_authors.avatarUrl: /avatars/3ed9473aee23d99f4ee949d3705089ea.svg
paper_authors.fullname: Yingbo Zhou
paper_authors.type: user
paper_authors.user: yingbozhou
paper_authors._id: 64fa7e88bb362cbf2fd324a2
paper_authors.name: Shafiq Joty
paper_authors._id: 64fa7e88bb362cbf2fd324a3
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T12:01:15.727Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_title: XGen-7B Technical Report
paper_summary: Large Language Models (LLMs) have become ubiquitous across various domains,
  transforming the way we interact with information and conduct research.
  However, most high-performing LLMs remain confined behind proprietary walls,
  hindering scientific progress. Most open-source LLMs, on the other hand, are
  limited in their ability to support longer sequence lengths, which is a key
  requirement for many tasks that require inference over an input context. To
  address this, we have trained XGen, a series of 7B parameter models on up to 8K
  sequence length for up to 1.5T tokens. We have also finetuned the XGen models
  on public-domain instructional data, creating their instruction-tuned
  counterparts (XGen-Inst). We open-source our models for both research
  advancements and commercial applications. Our evaluation on standard benchmarks
  shows that XGen models achieve comparable or better results when compared with
  state-of-the-art open-source LLMs. Our targeted evaluation on long sequence
  modeling tasks shows the benefits of our 8K-sequence models over 2K-sequence
  open-source LLMs.
paper_ai_keywords: parameter-efficient fine-tuning denoising autoencoders sequence lengths instructional data instruction-tuned long sequence modeling benchmarks sequence models