title: Text Rendering Strategies for Pixel Language Models
summary: Pixel-based language models process text rendered as images, which allows
  them to handle any script, making them a promising approach to open vocabulary
  language modelling. However, recent approaches use text renderers that produce
  a large set of almost-equivalent input patches, which may prove sub-optimal for
  downstream tasks, due to redundancy in the input representations. In this
  paper, we investigate four approaches to rendering text in the PIXEL model
  (Rust et al., 2023), and find that simple character bigram rendering brings
  improved performance on sentence-level tasks without compromising performance
  on token-level or multilingual tasks. This new rendering strategy also makes it
  possible to train a more compact model with only 22M parameters that performs
  on par with the original 86M parameter model. Our analyses show that character
  bigram rendering leads to a consistently better model but with an anisotropic
  patch embedding space, driven by a patch frequency bias, highlighting the
  connections between image patch- and tokenization-based language models.
paper_authors._id: 654307824a68d5aeba5b67cc
paper_authors.name: Jonas F. Lotz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:54:56.838Z
paper_authors._id: 627e4a0b1d60305a5c8fc9fd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/627e4a0b1d60305a5c8fc9fd/zbkmD2k6bILHjWxgXnsR7.png
paper_authors.fullname: Jonas F. Lotz
paper_authors.type: user
paper_authors.user: jflotz
paper_authors._id: 654307824a68d5aeba5b67cd
paper_authors.name: Elizabeth Salesky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:55:05.386Z
paper_authors._id: 60521a52d8c0f8f1ffaaf2cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659018192837-60521a52d8c0f8f1ffaaf2cc.jpeg
paper_authors.fullname: Elizabeth Salesky
paper_authors.type: user
paper_authors.user: esalesky
paper_authors._id: 654307824a68d5aeba5b67ce
paper_authors.name: Phillip Rust
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:55:13.328Z
paper_authors._id: 6079adbad2cd8c150e6ae05c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648912870656-6079adbad2cd8c150e6ae05c.jpeg
paper_authors.fullname: Phillip Rust
paper_authors.type: user
paper_authors.user: plip
paper_authors._id: 654307824a68d5aeba5b67cf
paper_authors.name: Desmond Elliott
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:55:19.519Z
paper_authors._id: 6285f66a7cc3b7bc1b8e7b8e
paper_authors.avatarUrl: /avatars/984ae22db7cc885591bc0b5bceffdfbd.svg
paper_authors.fullname: Desmond Elliott
paper_authors.type: user
paper_authors.user: elliottd
paper_title: Text Rendering Strategies for Pixel Language Models
paper_summary: Pixel-based language models process text rendered as images, which allows
  them to handle any script, making them a promising approach to open vocabulary
  language modelling. However, recent approaches use text renderers that produce
  a large set of almost-equivalent input patches, which may prove sub-optimal for
  downstream tasks, due to redundancy in the input representations. In this
  paper, we investigate four approaches to rendering text in the PIXEL model
  (Rust et al., 2023), and find that simple character bigram rendering brings
  improved performance on sentence-level tasks without compromising performance
  on token-level or multilingual tasks. This new rendering strategy also makes it
  possible to train a more compact model with only 22M parameters that performs
  on par with the original 86M parameter model. Our analyses show that character
  bigram rendering leads to a consistently better model but with an anisotropic
  patch embedding space, driven by a patch frequency bias, highlighting the
  connections between image patch- and tokenization-based language models.
paper_ai_summary: Character bigram rendering improves performance in pixel-based language models, enabling more compact models while maintaining equivalent performance across sentence, token, and multilingual tasks.
paper_ai_keywords: pixel-based language models text renderers input patches character bigram rendering sentence-level tasks token-level tasks multilingual tasks patch embedding space tokenization-based language models