title: Efficient Track Anything
summary: Segment Anything Model 2 (SAM 2) has emerged as a powerful tool for video
  object segmentation and tracking anything. Key components of SAM 2 that drive
  the impressive video object segmentation performance include a large multistage
  image encoder for frame feature extraction and a memory mechanism that stores
  memory contexts from past frames to help current frame segmentation. The high
  computation complexity of multistage image encoder and memory module has
  limited its applications in real-world tasks, e.g., video object segmentation
  on mobile devices. To address this limitation, we propose EfficientTAMs,
  lightweight track anything models that produce high-quality results with low
  latency and model size. Our idea is based on revisiting the plain,
  nonhierarchical Vision Transformer (ViT) as an image encoder for video object
  segmentation, and introducing an efficient memory module, which reduces the
  complexity for both frame feature extraction and memory computation for current
  frame segmentation. We take vanilla lightweight ViTs and efficient memory
  module to build EfficientTAMs, and train the models on SA-1B and SA-V datasets
  for video object segmentation and track anything tasks. We evaluate on multiple
  video segmentation benchmarks including semi-supervised VOS and promptable
  video segmentation, and find that our proposed EfficientTAM with vanilla ViT
  perform comparably to SAM 2 model (HieraB+SAM 2) with ~2x speedup on A100 and
  ~2.4x parameter reduction. On segment anything image tasks, our EfficientTAMs
  also perform favorably over original SAM with ~20x speedup on A100 and ~20x
  parameter reduction. On mobile devices such as iPhone 15 Pro Max, our
  EfficientTAMs can run at ~10 FPS for performing video object segmentation with
  reasonable quality, highlighting the capability of small models for on-device
  video object segmentation applications.
paper_authors._id: 674e953b1ff3268f006ab9ea
paper_authors.name: Yunyang Xiong
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-03T13:05:39.894Z
paper_authors._id: 65304b62e7535baecd85d080
paper_authors.avatarUrl: /avatars/6e546c7d1414bd92c5a7c8d8c404de92.svg
paper_authors.fullname: Yunyang Xiong
paper_authors.type: user
paper_authors.user: yunyangx
paper_authors._id: 674e953b1ff3268f006ab9eb
paper_authors.name: Chong Zhou
paper_authors._id: 674e953b1ff3268f006ab9ec
paper_authors.name: Xiaoyu Xiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:15:41.566Z
paper_authors._id: 6350675859bfa9a85d40c28b
paper_authors.avatarUrl: /avatars/13fdc2b86f7f8191903bd65e6112acc8.svg
paper_authors.fullname: Xiaoyu Xiang
paper_authors.type: user
paper_authors.user: mukosame
paper_authors._id: 674e953b1ff3268f006ab9ed
paper_authors.name: Lemeng Wu
paper_authors._id: 674e953b1ff3268f006ab9ee
paper_authors.name: Chenchen Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:15:27.374Z
paper_authors._id: 66f6e80e7db9927533cbd3e1
paper_authors.avatarUrl: /avatars/f169a3cb4f48781e38cc111a35741d2d.svg
paper_authors.fullname: Chenchen Zhu
paper_authors.type: user
paper_authors.user: zcckernel
paper_authors._id: 674e953b1ff3268f006ab9ef
paper_authors.name: Zechun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:15:32.959Z
paper_authors._id: 660f893bae89429c07a32cdb
paper_authors.avatarUrl: /avatars/27442b1dab58114cfe10220c040c1156.svg
paper_authors.fullname: Zechun Liu
paper_authors.type: user
paper_authors.user: zechunliu
paper_authors._id: 674e953b1ff3268f006ab9f0
paper_authors.name: Saksham Suri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:15:50.251Z
paper_authors._id: 645417e98617183806210cfc
paper_authors.avatarUrl: /avatars/7a3ba5b0c38878db51da7091fed15f99.svg
paper_authors.fullname: Saksham Suri
paper_authors.type: user
paper_authors.user: sakshams
paper_authors._id: 674e953b1ff3268f006ab9f1
paper_authors.name: Balakrishnan Varadarajan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:14:54.717Z
paper_authors._id: 6570032714fa8cfccd39c6ad
paper_authors.avatarUrl: /avatars/f4b6140e21db5d18241dcc9b2e94ae33.svg
paper_authors.fullname: Balakrishnan Varadarajan
paper_authors.type: user
paper_authors.user: balakv
paper_authors._id: 674e953b1ff3268f006ab9f2
paper_authors.name: Ramya Akula
paper_authors._id: 674e953b1ff3268f006ab9f3
paper_authors.name: Forrest Iandola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:15:07.365Z
paper_authors._id: 5f1753f4925b9863e28ad4de
paper_authors.avatarUrl: /avatars/e56d25a40f2c0e4e6696b9738877fce8.svg
paper_authors.fullname: Forrest Iandola
paper_authors.type: user
paper_authors.user: forresti
paper_authors._id: 674e953b1ff3268f006ab9f4
paper_authors.name: Raghuraman Krishnamoorthi
paper_authors._id: 674e953b1ff3268f006ab9f5
paper_authors.name: Bilge Soran
paper_authors._id: 674e953b1ff3268f006ab9f6
paper_authors.name: Vikas Chandra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:16:17.253Z
paper_authors._id: 6566681f7b5ed0735812af32
paper_authors.avatarUrl: /avatars/424905abb2973954b0850c592743b6fb.svg
paper_authors.fullname: Vikas Chandra
paper_authors.type: user
paper_authors.user: vchandra
paper_title: Efficient Track Anything
paper_summary: Segment Anything Model 2 (SAM 2) has emerged as a powerful tool for video
  object segmentation and tracking anything. Key components of SAM 2 that drive
  the impressive video object segmentation performance include a large multistage
  image encoder for frame feature extraction and a memory mechanism that stores
  memory contexts from past frames to help current frame segmentation. The high
  computation complexity of multistage image encoder and memory module has
  limited its applications in real-world tasks, e.g., video object segmentation
  on mobile devices. To address this limitation, we propose EfficientTAMs,
  lightweight track anything models that produce high-quality results with low
  latency and model size. Our idea is based on revisiting the plain,
  nonhierarchical Vision Transformer (ViT) as an image encoder for video object
  segmentation, and introducing an efficient memory module, which reduces the
  complexity for both frame feature extraction and memory computation for current
  frame segmentation. We take vanilla lightweight ViTs and efficient memory
  module to build EfficientTAMs, and train the models on SA-1B and SA-V datasets
  for video object segmentation and track anything tasks. We evaluate on multiple
  video segmentation benchmarks including semi-supervised VOS and promptable
  video segmentation, and find that our proposed EfficientTAM with vanilla ViT
  perform comparably to SAM 2 model (HieraB+SAM 2) with ~2x speedup on A100 and
  ~2.4x parameter reduction. On segment anything image tasks, our EfficientTAMs
  also perform favorably over original SAM with ~20x speedup on A100 and ~20x
  parameter reduction. On mobile devices such as iPhone 15 Pro Max, our
  EfficientTAMs can run at ~10 FPS for performing video object segmentation with
  reasonable quality, highlighting the capability of small models for on-device
  video object segmentation applications.
paper_ai_keywords: Vision Transformer (ViT) EfficientTAMs SA-1B dataset SA-V dataset semi-supervised VOS promptable video segmentation Hierarchical Vision Transformer (HieraB+SAM 2)