Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary
  Resolution
Visual data comes in various forms, ranging from small icons of just a few
pixels to long videos spanning hours. Existing multi-modal LLMs usually
standardize these diverse visual inputs to a fixed resolution for visual
encoders and yield similar numbers of tokens for LLMs. This approach is
non-optimal for multimodal understanding and inefficient for processing inputs
with long and short visual contents. To solve the problem, we propose Oryx, a
unified multimodal architecture for the spatial-temporal understanding of
images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to
seamlessly and efficiently process visual inputs with arbitrary spatial sizes
and temporal lengths through two core innovations: 1) a pre-trained OryxViT
model that can encode images at any resolution into LLM-friendly visual
representations; 2) a dynamic compressor module that supports 1x to 16x
compression on visual tokens by request. These design features enable Oryx to
accommodate extremely long visual contexts, such as videos, with lower
resolution and high compression while maintaining high recognition precision
for tasks like document understanding with native resolution and no
compression. Beyond the architectural improvements, enhanced data curation and
specialized training on long-context retrieval and spatial-aware data help Oryx
achieve strong capabilities in image, video, and 3D multimodal understanding
simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.
66ece4e1831eacfcad321d82
Zuyan Liu
claimed_verified
2024-09-20T07:45:51.806Z
64f001bfabd9fb1914398bd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64f001bfabd9fb1914398bd5/9teH82hkBI4csIz_WQh5q.jpeg
liuzuyan
user
Zuyan
66ece4e1831eacfcad321d83
Yuhao Dong
admin_assigned
2024-09-20T08:54:29.275Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
66ece4e1831eacfcad321d84
Ziwei Liu
admin_assigned
2024-09-20T08:54:35.459Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
66ece4e1831eacfcad321d85
Winston Hu
admin_assigned
2024-09-20T08:54:41.397Z
63673bb9d0ee6e2662be0ec1
/avatars/1b8976785d64bc4e3f7159ccdb7f06c5.svg
Qingqiao Hu
user
WinstonHu
66ece4e1831eacfcad321d86
Jiwen Lu
admin_assigned
2024-09-20T08:54:48.137Z
66c44203ea476bea05e9fcd7
/avatars/b061eebec609446e669f5ad6365959f9.svg
lu
user
jiwenlu
66ece4e1831eacfcad321d87
Yongming Rao
admin_assigned
2024-09-20T08:54:53.609Z
63e4865354f51ea342d45d78
/avatars/2e7eccc878751331ca8b282f53e38899.svg
Yongming Rao
user
raoyongming
Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary
  Resolution
Visual data comes in various forms, ranging from small icons of just a few
pixels to long videos spanning hours. Existing multi-modal LLMs usually
standardize these diverse visual inputs to a fixed resolution for visual
encoders and yield similar numbers of tokens for LLMs. This approach is
non-optimal for multimodal understanding and inefficient for processing inputs
with long and short visual contents. To solve the problem, we propose Oryx, a
unified multimodal architecture for the spatial-temporal understanding of
images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to
seamlessly and efficiently process visual inputs with arbitrary spatial sizes
and temporal lengths through two core innovations: 1) a pre-trained OryxViT
model that can encode images at any resolution into LLM-friendly visual
representations; 2) a dynamic compressor module that supports 1x to 16x
compression on visual tokens by request. These design features enable Oryx to
accommodate extremely long visual contexts, such as videos, with lower
resolution and high compression while maintaining high recognition precision
for tasks like document understanding with native resolution and no
compression. Beyond the architectural improvements, enhanced data curation and
specialized training on long-context retrieval and spatial-aware data help Oryx
achieve strong capabilities in image, video, and 3D multimodal understanding
simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.
Oryx, a unified multimodal architecture, uses a pre-trained OryxViT model and a dynamic compressor to efficiently process diverse visual inputs of varying resolutions and lengths, enhancing spatial-temporal understanding and multimodal capabilities.
Oryx OryxViT dynamic compressor multimodal spatial-temporal understanding pre-trained visual representations long-context retrieval spatial-aware data