title: Subobject-level Image Tokenization
summary: Transformer-based vision models typically tokenize images into fixed-size
  square patches as input units, which lacks the adaptability to image content
  and overlooks the inherent pixel grouping structure. Inspired by the subword
  tokenization widely adopted in language models, we propose an image tokenizer
  at a subobject level, where the subobjects are represented by semantically
  meaningful image segments obtained by segmentation models (e.g., segment
  anything models). To implement a learning system based on subobject
  tokenization, we first introduced a Sequence-to-sequence AutoEncoder (SeqAE) to
  compress subobject segments of varying sizes and shapes into compact embedding
  vectors, then fed the subobject embeddings into a large language model for
  vision language learning. Empirical results demonstrated that our
  subobject-level tokenization significantly facilitates efficient learning of
  translating images into object and attribute descriptions compared to the
  traditional patch-level tokenization. Codes and models will be open-sourced at
  https://github.com/ChenDelong1999/subobjects.
paper_authors._id: 65d81106c790f52e937f94eb
paper_authors.name: Delong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:56:34.973Z
paper_authors._id: 630491107424d937fa3258be
paper_authors.avatarUrl: /avatars/b8bd81bc8544674ee26b78702afdb87c.svg
paper_authors.fullname: Delong Chen
paper_authors.type: user
paper_authors.user: chendelong
paper_authors._id: 65d81106c790f52e937f94ec
paper_authors.name: Samuel Cahyawijaya
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:56:40.946Z
paper_authors._id: 5f5c4d49e56d546cd623309b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1599884443706-noauth.jpeg
paper_authors.fullname: Samuel Cahyawijaya
paper_authors.type: user
paper_authors.user: samuelcahyawijaya
paper_authors._id: 65d81106c790f52e937f94ed
paper_authors.name: Jianfeng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:56:47.609Z
paper_authors._id: 62f1db70a39cf6f5c63fc5e5
paper_authors.avatarUrl: /avatars/ff4a0f4f4ddab4cc87cf3b3abea0f232.svg
paper_authors.fullname: Jianfeng Liu
paper_authors.type: user
paper_authors.user: docacola
paper_authors._id: 65d81106c790f52e937f94ee
paper_authors.name: Baoyuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-23T12:56:55.119Z
paper_authors._id: 64b80f042dde7191bf229d85
paper_authors.avatarUrl: /avatars/34ea357b9c42ac8c40133936992e0005.svg
paper_authors.fullname: Baoyuan Wang
paper_authors.type: user
paper_authors.user: baoyuanw
paper_authors._id: 65d81106c790f52e937f94ef
paper_authors.name: Pascale Fung
paper_title: Subobject-level Image Tokenization
paper_summary: Transformer-based vision models typically tokenize images into fixed-size
  square patches as input units, which lacks the adaptability to image content
  and overlooks the inherent pixel grouping structure. Inspired by the subword
  tokenization widely adopted in language models, we propose an image tokenizer
  at a subobject level, where the subobjects are represented by semantically
  meaningful image segments obtained by segmentation models (e.g., segment
  anything models). To implement a learning system based on subobject
  tokenization, we first introduced a Sequence-to-sequence AutoEncoder (SeqAE) to
  compress subobject segments of varying sizes and shapes into compact embedding
  vectors, then fed the subobject embeddings into a large language model for
  vision language learning. Empirical results demonstrated that our
  subobject-level tokenization significantly facilitates efficient learning of
  translating images into object and attribute descriptions compared to the
  traditional patch-level tokenization. Codes and models will be open-sourced at
  https://github.com/ChenDelong1999/subobjects.
paper_ai_keywords: Transformer-based vision models tokenization images fixed-size square patches subword tokenization language models image tokenizer subobject level semantically meaningful image segments segmentation models Sequence-to-sequence AutoEncoder (SeqAE) compact embedding vectors large language model vision language learning subobject-level tokenization patch-level tokenization