Tracking through Containers and Occluders in the Wild
Tracking objects with persistence in cluttered and dynamic environments
remains a difficult challenge for computer vision systems. In this paper, we
introduce TCOW, a new benchmark and model for visual tracking
through heavy occlusion and containment. We set up a task where the goal is to,
given a video sequence, segment both the projected extent of the target object,
as well as the surrounding container or occluder whenever one exists. To study
this task, we create a mixture of synthetic and annotated real datasets to
support both supervised learning and structured evaluation of model performance
under various forms of task variation, such as moving or nested containment. We
evaluate two recent transformer-based video models and find that while they can
be surprisingly capable of tracking targets under certain settings of task
variation, there remains a considerable performance gap before we can claim a
tracking model to have acquired a true notion of object permanence.
64547b27c13cdb83f10f445e
Basile Van Hoorick
admin_assigned
2023-06-01T14:50:57.671Z
631f9e1cb6628770f6f2e7fb
/avatars/2f5a024e240ff9446b19054f597e6e91.svg
Basile Van Hoorick
user
basilevh
64547b27c13cdb83f10f445f
Pavel Tokmakov
claimed_verified
2024-01-22T21:12:10.052Z
65aeab2201b6f61e24323339
/avatars/2c119173c0871f59df108f9bb4471954.svg
Pavel Tokmakov
user
ptokmakov
64547b27c13cdb83f10f4460
Simon Stent
64547b27c13cdb83f10f4461
Jie Li
64547b27c13cdb83f10f4462
Carl Vondrick
admin_assigned
2023-06-01T14:51:53.325Z
63a4bfcbcde2b28f82a2c50c
/avatars/9d978a6f0caf6ad92ec4138a84a39e53.svg
Carl Vondrick
user
cvondrick
Tracking through Containers and Occluders in the Wild
Tracking objects with persistence in cluttered and dynamic environments
remains a difficult challenge for computer vision systems. In this paper, we
introduce TCOW, a new benchmark and model for visual tracking
through heavy occlusion and containment. We set up a task where the goal is to,
given a video sequence, segment both the projected extent of the target object,
as well as the surrounding container or occluder whenever one exists. To study
this task, we create a mixture of synthetic and annotated real datasets to
support both supervised learning and structured evaluation of model performance
under various forms of task variation, such as moving or nested containment. We
evaluate two recent transformer-based video models and find that while they can
be surprisingly capable of tracking targets under certain settings of task
variation, there remains a considerable performance gap before we can claim a
tracking model to have acquired a true notion of object permanence.
TCOW is a benchmark and model for visual tracking in cluttered environments with heavy occlusions and containment, using a mixture of synthetic and real datasets to evaluate transformer-based video models' performance in understanding object permanence.
transformer-based video models object permanence