Beyond Language Models: Byte Models are Digital World Simulators
Traditional deep learning often overlooks bytes, the basic units of the
digital world, where all forms of information and operations are encoded and
manipulated in binary format. Inspired by the success of next token prediction
in natural language processing, we introduce bGPT, a model with next byte
prediction to simulate the digital world. bGPT matches specialized models in
performance across various modalities, including text, audio, and images, and
offers new possibilities for predicting, simulating, and diagnosing algorithm
or hardware behaviour. It has almost flawlessly replicated the process of
converting symbolic music data, achieving a low error rate of 0.0011 bits per
byte in converting ABC notation to MIDI format. In addition, bGPT demonstrates
exceptional capabilities in simulating CPU behaviour, with an accuracy
exceeding 99.99% in executing various operations. Leveraging next byte
prediction, models like bGPT can directly learn from vast binary data,
effectively simulating the intricate patterns of the digital world.
65e15544ae0a6579208a7048
Shangda Wu
65e15544ae0a6579208a7049
Xu Tan
admin_assigned
2024-03-01T10:40:18.447Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
65e15544ae0a6579208a704a
Zili Wang
admin_assigned
2024-03-01T10:41:11.865Z
6410998db27543634e3aacc3
https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
Zili Wang
user
MarkWang
65e15544ae0a6579208a704b
Rui Wang
65e15544ae0a6579208a704c
Xiaobing Li
65e15544ae0a6579208a704d
Maosong Sun
Beyond Language Models: Byte Models are Digital World Simulators
Traditional deep learning often overlooks bytes, the basic units of the
digital world, where all forms of information and operations are encoded and
manipulated in binary format. Inspired by the success of next token prediction
in natural language processing, we introduce bGPT, a model with next byte
prediction to simulate the digital world. bGPT matches specialized models in
performance across various modalities, including text, audio, and images, and
offers new possibilities for predicting, simulating, and diagnosing algorithm
or hardware behaviour. It has almost flawlessly replicated the process of
converting symbolic music data, achieving a low error rate of 0.0011 bits per
byte in converting ABC notation to MIDI format. In addition, bGPT demonstrates
exceptional capabilities in simulating CPU behaviour, with an accuracy
exceeding 99.99% in executing various operations. Leveraging next byte
prediction, models like bGPT can directly learn from vast binary data,
effectively simulating the intricate patterns of the digital world.
bGPT, a model using next byte prediction, simulates various digital processes with high accuracy, matching specialized models in performance and offering insights into algorithm and hardware behavior.
bGPT next byte prediction ABC notation MIDI format digital world symbolic music data CPU behaviour