title: Tracking through Containers and Occluders in the Wild
summary: Tracking objects with persistence in cluttered and dynamic environments
  remains a difficult challenge for computer vision systems. In this paper, we
  introduce TCOW, a new benchmark and model for visual tracking
  through heavy occlusion and containment. We set up a task where the goal is to,
  given a video sequence, segment both the projected extent of the target object,
  as well as the surrounding container or occluder whenever one exists. To study
  this task, we create a mixture of synthetic and annotated real datasets to
  support both supervised learning and structured evaluation of model performance
  under various forms of task variation, such as moving or nested containment. We
  evaluate two recent transformer-based video models and find that while they can
  be surprisingly capable of tracking targets under certain settings of task
  variation, there remains a considerable performance gap before we can claim a
  tracking model to have acquired a true notion of object permanence.
paper_authors._id: 64547b27c13cdb83f10f445e
paper_authors.name: Basile Van Hoorick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-01T14:50:57.671Z
paper_authors._id: 631f9e1cb6628770f6f2e7fb
paper_authors.avatarUrl: /avatars/2f5a024e240ff9446b19054f597e6e91.svg
paper_authors.fullname: Basile Van Hoorick
paper_authors.type: user
paper_authors.user: basilevh
paper_authors._id: 64547b27c13cdb83f10f445f
paper_authors.name: Pavel Tokmakov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T21:12:10.052Z
paper_authors._id: 65aeab2201b6f61e24323339
paper_authors.avatarUrl: /avatars/2c119173c0871f59df108f9bb4471954.svg
paper_authors.fullname: Pavel Tokmakov
paper_authors.type: user
paper_authors.user: ptokmakov
paper_authors._id: 64547b27c13cdb83f10f4460
paper_authors.name: Simon Stent
paper_authors._id: 64547b27c13cdb83f10f4461
paper_authors.name: Jie Li
paper_authors._id: 64547b27c13cdb83f10f4462
paper_authors.name: Carl Vondrick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-01T14:51:53.325Z
paper_authors._id: 63a4bfcbcde2b28f82a2c50c
paper_authors.avatarUrl: /avatars/9d978a6f0caf6ad92ec4138a84a39e53.svg
paper_authors.fullname: Carl Vondrick
paper_authors.type: user
paper_authors.user: cvondrick
paper_title: Tracking through Containers and Occluders in the Wild
paper_summary: Tracking objects with persistence in cluttered and dynamic environments
  remains a difficult challenge for computer vision systems. In this paper, we
  introduce TCOW, a new benchmark and model for visual tracking
  through heavy occlusion and containment. We set up a task where the goal is to,
  given a video sequence, segment both the projected extent of the target object,
  as well as the surrounding container or occluder whenever one exists. To study
  this task, we create a mixture of synthetic and annotated real datasets to
  support both supervised learning and structured evaluation of model performance
  under various forms of task variation, such as moving or nested containment. We
  evaluate two recent transformer-based video models and find that while they can
  be surprisingly capable of tracking targets under certain settings of task
  variation, there remains a considerable performance gap before we can claim a
  tracking model to have acquired a true notion of object permanence.
paper_ai_keywords: TCOW transformer-based video models object permanence