title: Tracking Anything with Decoupled Video Segmentation
summary: Training data for video segmentation are expensive to annotate. This impedes
  extensions of end-to-end algorithms to new video segmentation tasks, especially
  in large-vocabulary settings. To 'track anything' without training on video
  data for every individual task, we develop a decoupled video segmentation
  approach (DEVA), composed of task-specific image-level segmentation and
  class/task-agnostic bi-directional temporal propagation. Due to this design, we
  only need an image-level model for the target task (which is cheaper to train)
  and a universal temporal propagation model which is trained once and
  generalizes across tasks. To effectively combine these two modules, we use
  bi-directional propagation for (semi-)online fusion of segmentation hypotheses
  from different frames to generate a coherent segmentation. We show that this
  decoupled formulation compares favorably to end-to-end approaches in several
  data-scarce tasks including large-vocabulary video panoptic segmentation,
  open-world video segmentation, referring video segmentation, and unsupervised
  video object segmentation. Code is available at:
  https://hkchengrex.github.io/Tracking-Anything-with-DEVA
paper_authors._id: 64faba2c8c21ebb3db635594
paper_authors.name: Ho Kei Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:39:09.024Z
paper_authors._id: 63041b541dd5d3c62486c294
paper_authors.avatarUrl: /avatars/a5286d562f7b9082730f760e66c3bf29.svg
paper_authors.fullname: Ho Kei Cheng
paper_authors.type: user
paper_authors.user: hkchengrex
paper_authors._id: 64faba2c8c21ebb3db635595
paper_authors.name: Seoung Wug Oh
paper_authors._id: 64faba2c8c21ebb3db635596
paper_authors.name: Brian Price
paper_authors._id: 64faba2c8c21ebb3db635597
paper_authors.name: Alexander Schwing
paper_authors._id: 64faba2c8c21ebb3db635598
paper_authors.name: Joon-Young Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:39:49.340Z
paper_authors._id: 64b76c7990b38df83381824b
paper_authors.avatarUrl: /avatars/0a5bec2ea480fb3f43c4b24d59c50e81.svg
paper_authors.fullname: Joon-Young Lee
paper_authors.type: user
paper_authors.user: joonyounglee
paper_title: Tracking Anything with Decoupled Video Segmentation
paper_summary: Training data for video segmentation are expensive to annotate. This impedes
  extensions of end-to-end algorithms to new video segmentation tasks, especially
  in large-vocabulary settings. To 'track anything' without training on video
  data for every individual task, we develop a decoupled video segmentation
  approach (DEVA), composed of task-specific image-level segmentation and
  class/task-agnostic bi-directional temporal propagation. Due to this design, we
  only need an image-level model for the target task (which is cheaper to train)
  and a universal temporal propagation model which is trained once and
  generalizes across tasks. To effectively combine these two modules, we use
  bi-directional propagation for (semi-)online fusion of segmentation hypotheses
  from different frames to generate a coherent segmentation. We show that this
  decoupled formulation compares favorably to end-to-end approaches in several
  data-scarce tasks including large-vocabulary video panoptic segmentation,
  open-world video segmentation, referring video segmentation, and unsupervised
  video object segmentation. Code is available at:
  https://hkchengrex.github.io/Tracking-Anything-with-DEVA
paper_ai_keywords: decoupled video segmentation task-specific image-level segmentation class/task-agnostic bi-directional temporal propagation bi-directional propagation (semi-)online fusion segmentation hypotheses large-vocabulary video panoptic segmentation open-world video segmentation referring video segmentation unsupervised video object segmentation