title: Beyond Language Models: Byte Models are Digital World Simulators
summary: Traditional deep learning often overlooks bytes, the basic units of the
  digital world, where all forms of information and operations are encoded and
  manipulated in binary format. Inspired by the success of next token prediction
  in natural language processing, we introduce bGPT, a model with next byte
  prediction to simulate the digital world. bGPT matches specialized models in
  performance across various modalities, including text, audio, and images, and
  offers new possibilities for predicting, simulating, and diagnosing algorithm
  or hardware behaviour. It has almost flawlessly replicated the process of
  converting symbolic music data, achieving a low error rate of 0.0011 bits per
  byte in converting ABC notation to MIDI format. In addition, bGPT demonstrates
  exceptional capabilities in simulating CPU behaviour, with an accuracy
  exceeding 99.99% in executing various operations. Leveraging next byte
  prediction, models like bGPT can directly learn from vast binary data,
  effectively simulating the intricate patterns of the digital world.
paper_authors._id: 65e15544ae0a6579208a7048
paper_authors.name: Shangda Wu
paper_authors._id: 65e15544ae0a6579208a7049
paper_authors.name: Xu Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:40:18.447Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 65e15544ae0a6579208a704a
paper_authors.name: Zili Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-01T10:41:11.865Z
paper_authors._id: 6410998db27543634e3aacc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
paper_authors.fullname: Zili Wang
paper_authors.type: user
paper_authors.user: MarkWang
paper_authors._id: 65e15544ae0a6579208a704b
paper_authors.name: Rui Wang
paper_authors._id: 65e15544ae0a6579208a704c
paper_authors.name: Xiaobing Li
paper_authors._id: 65e15544ae0a6579208a704d
paper_authors.name: Maosong Sun
paper_title: Beyond Language Models: Byte Models are Digital World Simulators
paper_summary: Traditional deep learning often overlooks bytes, the basic units of the
  digital world, where all forms of information and operations are encoded and
  manipulated in binary format. Inspired by the success of next token prediction
  in natural language processing, we introduce bGPT, a model with next byte
  prediction to simulate the digital world. bGPT matches specialized models in
  performance across various modalities, including text, audio, and images, and
  offers new possibilities for predicting, simulating, and diagnosing algorithm
  or hardware behaviour. It has almost flawlessly replicated the process of
  converting symbolic music data, achieving a low error rate of 0.0011 bits per
  byte in converting ABC notation to MIDI format. In addition, bGPT demonstrates
  exceptional capabilities in simulating CPU behaviour, with an accuracy
  exceeding 99.99% in executing various operations. Leveraging next byte
  prediction, models like bGPT can directly learn from vast binary data,
  effectively simulating the intricate patterns of the digital world.
paper_ai_keywords: bGPT next byte prediction symbolic music data ABC notation MIDI format CPU behaviour