title: PixelWorld: Towards Perceiving Everything as Pixels
summary: Existing foundation models typically process visual input as pixels and
  textual input as tokens, a paradigm that contrasts with human perception, where
  both modalities are processed in a unified manner. With the rise of embodied
  and agentic AI, where inputs primarily come from camera pixels, the need for a
  unified perception framework becomes increasingly evident. In this paper, we
  propose to unify all modalities (text, tables, code, diagrams, images, etc) as
  pixel inputs, i.e. "Perceive Everything as Pixels" (PEAP). We introduce
  PixelWorld, a novel evaluation suite that unifies all the mentioned modalities
  into pixel space to gauge the existing models' performance. Our findings show
  that (1) PEAP outperforms baseline with token-based input in multimodal
  datasets, benefiting from unified input for better disambiguation, (2)
  significant declines in reasoning and coding capabilities across all models
  when processing pixel-based input, underscoring the need to enhance foundation
  models' perceptual abilities, (3) larger models can maintain strong performance
  on non-reasoning tasks under PEAP, while smaller models like Phi-3.5-V suffer
  significant performance degradation, (4) the attention pattern of PEAP is
  highly aligned with text token input, (5) PEAP can be accelerated significantly
  by exploiting the spatial sparsity. We conclude that the existing frontier
  models are competent in pixel perception, however, there is still headroom for
  improvement. Our code, dataset will be released upon acceptance.
paper_authors._id: 67a044d1af1b65169565354c
paper_authors.name: Zhiheng Lyu
paper_authors._id: 67a044d1af1b65169565354d
paper_authors.name: Xueguang Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T09:28:04.822Z
paper_authors._id: 5ec82854968f6028e0559f70
paper_authors.avatarUrl: /avatars/45b58d912f7d00cb351947cd79d5eeb4.svg
paper_authors.fullname: Xueguang Ma
paper_authors.type: user
paper_authors.user: MrLight
paper_authors._id: 67a044d1af1b65169565354e
paper_authors.name: Wenhu Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-03T04:23:47.571Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: PixelWorld: Towards Perceiving Everything as Pixels
paper_summary: Existing foundation models typically process visual input as pixels and
  textual input as tokens, a paradigm that contrasts with human perception, where
  both modalities are processed in a unified manner. With the rise of embodied
  and agentic AI, where inputs primarily come from camera pixels, the need for a
  unified perception framework becomes increasingly evident. In this paper, we
  propose to unify all modalities (text, tables, code, diagrams, images, etc) as
  pixel inputs, i.e. "Perceive Everything as Pixels" (PEAP). We introduce
  PixelWorld, a novel evaluation suite that unifies all the mentioned modalities
  into pixel space to gauge the existing models' performance. Our findings show
  that (1) PEAP outperforms baseline with token-based input in multimodal
  datasets, benefiting from unified input for better disambiguation, (2)
  significant declines in reasoning and coding capabilities across all models
  when processing pixel-based input, underscoring the need to enhance foundation
  models' perceptual abilities, (3) larger models can maintain strong performance
  on non-reasoning tasks under PEAP, while smaller models like Phi-3.5-V suffer
  significant performance degradation, (4) the attention pattern of PEAP is
  highly aligned with text token input, (5) PEAP can be accelerated significantly
  by exploiting the spatial sparsity. We conclude that the existing frontier
  models are competent in pixel perception, however, there is still headroom for
  improvement. Our code, dataset will be released upon acceptance.
paper_ai_keywords: PixelWorld multimodal datasets reasoning and coding capabilities attention pattern spatial sparsity