title: Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary
    Resolution
summary: Visual data comes in various forms, ranging from small icons of just a few
  pixels to long videos spanning hours. Existing multi-modal LLMs usually
  standardize these diverse visual inputs to a fixed resolution for visual
  encoders and yield similar numbers of tokens for LLMs. This approach is
  non-optimal for multimodal understanding and inefficient for processing inputs
  with long and short visual contents. To solve the problem, we propose Oryx, a
  unified multimodal architecture for the spatial-temporal understanding of
  images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to
  seamlessly and efficiently process visual inputs with arbitrary spatial sizes
  and temporal lengths through two core innovations: 1) a pre-trained OryxViT
  model that can encode images at any resolution into LLM-friendly visual
  representations; 2) a dynamic compressor module that supports 1x to 16x
  compression on visual tokens by request. These design features enable Oryx to
  accommodate extremely long visual contexts, such as videos, with lower
  resolution and high compression while maintaining high recognition precision
  for tasks like document understanding with native resolution and no
  compression. Beyond the architectural improvements, enhanced data curation and
  specialized training on long-context retrieval and spatial-aware data help Oryx
  achieve strong capabilities in image, video, and 3D multimodal understanding
  simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.
paper_authors._id: 66ece4e1831eacfcad321d82
paper_authors.name: Zuyan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:45:51.806Z
paper_authors._id: 64f001bfabd9fb1914398bd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f001bfabd9fb1914398bd5/9teH82hkBI4csIz_WQh5q.jpeg
paper_authors.fullname: liuzuyan
paper_authors.type: user
paper_authors.user: Zuyan
paper_authors._id: 66ece4e1831eacfcad321d83
paper_authors.name: Yuhao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:54:29.275Z
paper_authors._id: 652965773a416e1f2173443b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
paper_authors.fullname: Yuhao Dong
paper_authors.type: user
paper_authors.user: THUdyh
paper_authors._id: 66ece4e1831eacfcad321d84
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:54:35.459Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 66ece4e1831eacfcad321d85
paper_authors.name: Winston Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:54:41.397Z
paper_authors._id: 63673bb9d0ee6e2662be0ec1
paper_authors.avatarUrl: /avatars/1b8976785d64bc4e3f7159ccdb7f06c5.svg
paper_authors.fullname: Qingqiao Hu
paper_authors.type: user
paper_authors.user: WinstonHu
paper_authors._id: 66ece4e1831eacfcad321d86
paper_authors.name: Jiwen Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:54:48.137Z
paper_authors._id: 66c44203ea476bea05e9fcd7
paper_authors.avatarUrl: /avatars/b061eebec609446e669f5ad6365959f9.svg
paper_authors.fullname: lu
paper_authors.type: user
paper_authors.user: jiwenlu
paper_authors._id: 66ece4e1831eacfcad321d87
paper_authors.name: Yongming Rao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:54:53.609Z
paper_authors._id: 63e4865354f51ea342d45d78
paper_authors.avatarUrl: /avatars/2e7eccc878751331ca8b282f53e38899.svg
paper_authors.fullname: Yongming Rao
paper_authors.type: user
paper_authors.user: raoyongming
paper_title: Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary
    Resolution
paper_summary: Visual data comes in various forms, ranging from small icons of just a few
  pixels to long videos spanning hours. Existing multi-modal LLMs usually
  standardize these diverse visual inputs to a fixed resolution for visual
  encoders and yield similar numbers of tokens for LLMs. This approach is
  non-optimal for multimodal understanding and inefficient for processing inputs
  with long and short visual contents. To solve the problem, we propose Oryx, a
  unified multimodal architecture for the spatial-temporal understanding of
  images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to
  seamlessly and efficiently process visual inputs with arbitrary spatial sizes
  and temporal lengths through two core innovations: 1) a pre-trained OryxViT
  model that can encode images at any resolution into LLM-friendly visual
  representations; 2) a dynamic compressor module that supports 1x to 16x
  compression on visual tokens by request. These design features enable Oryx to
  accommodate extremely long visual contexts, such as videos, with lower
  resolution and high compression while maintaining high recognition precision
  for tasks like document understanding with native resolution and no
  compression. Beyond the architectural improvements, enhanced data curation and
  specialized training on long-context retrieval and spatial-aware data help Oryx
  achieve strong capabilities in image, video, and 3D multimodal understanding
  simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.
paper_ai_keywords: Oryx multimodal architecture spatial-temporal understanding OryxViT visual representations dynamic compressor module long visual contexts high recognition precision long-context retrieval spatial-aware data