Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
trained by the raw byte values (hence the name). It is compared to PerceiverAR and a Transformer baseline. In all tasks, it outperforms both and is competitive with models that use tokenizers to discretize the input. The ablation analysis shows the importance of having both local and global models.