General Object Foundation Model for Images and Videos at Scale
We present GLEE in this work, an object-level foundation model for locating
and identifying objects in images and videos. Through a unified framework, GLEE
accomplishes detection, segmentation, tracking, grounding, and identification
of arbitrary objects in the open world scenario for various object perception
tasks. Adopting a cohesive learning strategy, GLEE acquires knowledge from
diverse data sources with varying supervision levels to formulate general
object representations, excelling in zero-shot transfer to new data and tasks.
Specifically, we employ an image encoder, text encoder, and visual prompter to
handle multi-modal inputs, enabling to simultaneously solve various
object-centric downstream tasks while maintaining state-of-the-art performance.
Demonstrated through extensive training on over five million images from
diverse benchmarks, GLEE exhibits remarkable versatility and improved
generalization performance, efficiently tackling downstream tasks without the
need for task-specific adaptation. By integrating large volumes of
automatically labeled data, we further enhance its zero-shot generalization
capabilities. Additionally, GLEE is capable of being integrated into Large
Language Models, serving as a foundational model to provide universal
object-level information for multi-modal tasks. We hope that the versatility
and universality of our method will mark a significant step in the development
of efficient visual foundation models for AGI systems. The model and code will
be released at https://glee-vision.github.io .
657baf29f81f6b44b89fe494
Junfeng Wu
claimed_verified
2023-12-15T13:28:39.118Z
6569de56046899997ba5ac4e
/avatars/287ee4c36a9edf2540eed779fd7d41e2.svg
Junfeng Wu
user
Junfeng5
657baf29f81f6b44b89fe495
Yi Jiang
claimed_verified
2024-04-05T09:33:09.388Z
6344dcb1cd37e44d9ed46508
https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
Yi Jiang
user
JiangYi
657baf29f81f6b44b89fe496
Qihao Liu
admin_assigned
2023-12-15T13:15:59.280Z
639f1e519f1f2baab2f00d22
https://cdn-avatars.huggingface.co/v1/production/uploads/639f1e519f1f2baab2f00d22/pFjd51WZuVZ3A11rItvmk.jpeg
Qihao Liu
user
QHL067
657baf29f81f6b44b89fe497
Zehuan Yuan
657baf29f81f6b44b89fe498
Xiang Bai
admin_assigned
2023-12-15T13:16:24.948Z
641790e2f1e86908935d82a0
/avatars/ced7a137c6344c74b7ac0d5c84833fc8.svg
Xiang Bai
user
baixianger
657baf29f81f6b44b89fe499
Song Bai
General Object Foundation Model for Images and Videos at Scale
We present GLEE in this work, an object-level foundation model for locating
and identifying objects in images and videos. Through a unified framework, GLEE
accomplishes detection, segmentation, tracking, grounding, and identification
of arbitrary objects in the open world scenario for various object perception
tasks. Adopting a cohesive learning strategy, GLEE acquires knowledge from
diverse data sources with varying supervision levels to formulate general
object representations, excelling in zero-shot transfer to new data and tasks.
Specifically, we employ an image encoder, text encoder, and visual prompter to
handle multi-modal inputs, enabling to simultaneously solve various
object-centric downstream tasks while maintaining state-of-the-art performance.
Demonstrated through extensive training on over five million images from
diverse benchmarks, GLEE exhibits remarkable versatility and improved
generalization performance, efficiently tackling downstream tasks without the
need for task-specific adaptation. By integrating large volumes of
automatically labeled data, we further enhance its zero-shot generalization
capabilities. Additionally, GLEE is capable of being integrated into Large
Language Models, serving as a foundational model to provide universal
object-level information for multi-modal tasks. We hope that the versatility
and universality of our method will mark a significant step in the development
of efficient visual foundation models for AGI systems. The model and code will
be released at https://glee-vision.github.io .
GLEE is an object-level foundation model for tasks like detection, segmentation, tracking, and identification, demonstrating zero-shot transfer and state-of-the-art performance.
object-level foundation model detection segmentation tracking grounding identification open world scenario object perception tasks cohesive learning strategy image encoder text encoder visual prompter multi-modal inputs downstream tasks zero-shot transfer state-of-the-art performance general object representations generalization performance large volumes of automatically labeled data Large Language Models AGI systems