Efficient Track Anything
Segment Anything Model 2 (SAM 2) has emerged as a powerful tool for video
object segmentation and tracking anything. Key components of SAM 2 that drive
the impressive video object segmentation performance include a large multistage
image encoder for frame feature extraction and a memory mechanism that stores
memory contexts from past frames to help current frame segmentation. The high
computation complexity of multistage image encoder and memory module has
limited its applications in real-world tasks, e.g., video object segmentation
on mobile devices. To address this limitation, we propose EfficientTAMs,
lightweight track anything models that produce high-quality results with low
latency and model size. Our idea is based on revisiting the plain,
nonhierarchical Vision Transformer (ViT) as an image encoder for video object
segmentation, and introducing an efficient memory module, which reduces the
complexity for both frame feature extraction and memory computation for current
frame segmentation. We take vanilla lightweight ViTs and efficient memory
module to build EfficientTAMs, and train the models on SA-1B and SA-V datasets
for video object segmentation and track anything tasks. We evaluate on multiple
video segmentation benchmarks including semi-supervised VOS and promptable
video segmentation, and find that our proposed EfficientTAM with vanilla ViT
perform comparably to SAM 2 model (HieraB+SAM 2) with ~2x speedup on A100 and
~2.4x parameter reduction. On segment anything image tasks, our EfficientTAMs
also perform favorably over original SAM with ~20x speedup on A100 and ~20x
parameter reduction. On mobile devices such as iPhone 15 Pro Max, our
EfficientTAMs can run at ~10 FPS for performing video object segmentation with
reasonable quality, highlighting the capability of small models for on-device
video object segmentation applications.
674e953b1ff3268f006ab9ea
Yunyang Xiong
extracted_confirmed
2024-12-03T13:05:39.894Z
65304b62e7535baecd85d080
/avatars/6e546c7d1414bd92c5a7c8d8c404de92.svg
Yunyang Xiong
user
yunyangx
674e953b1ff3268f006ab9eb
Chong Zhou
claimed_verified
2025-04-30T07:57:50.686Z
642bfdbbc885078517154abb
https://cdn-avatars.huggingface.co/v1/production/uploads/642bfdbbc885078517154abb/lLmc8_3VnTyk4qOvl-omI.jpeg
Chong Zhou
user
chongzhou
674e953b1ff3268f006ab9ec
Xiaoyu Xiang
admin_assigned
2024-12-03T16:15:41.566Z
6350675859bfa9a85d40c28b
/avatars/13fdc2b86f7f8191903bd65e6112acc8.svg
Xiaoyu Xiang
user
mukosame
674e953b1ff3268f006ab9ed
Lemeng Wu
674e953b1ff3268f006ab9ee
Chenchen Zhu
admin_assigned
2024-12-03T16:15:27.374Z
66f6e80e7db9927533cbd3e1
/avatars/f169a3cb4f48781e38cc111a35741d2d.svg
Chenchen Zhu
user
zcckernel
674e953b1ff3268f006ab9ef
Zechun Liu
admin_assigned
2024-12-03T16:15:32.959Z
660f893bae89429c07a32cdb
/avatars/27442b1dab58114cfe10220c040c1156.svg
Zechun Liu
user
zechunliu
674e953b1ff3268f006ab9f0
Saksham Suri
admin_assigned
2024-12-03T16:15:50.251Z
645417e98617183806210cfc
/avatars/7a3ba5b0c38878db51da7091fed15f99.svg
Saksham Suri
user
sakshams
674e953b1ff3268f006ab9f1
Balakrishnan Varadarajan
admin_assigned
2024-12-03T16:14:54.717Z
6570032714fa8cfccd39c6ad
/avatars/f4b6140e21db5d18241dcc9b2e94ae33.svg
Balakrishnan Varadarajan
user
balakv
674e953b1ff3268f006ab9f2
Ramya Akula
674e953b1ff3268f006ab9f3
Forrest Iandola
admin_assigned
2024-12-03T16:15:07.365Z
5f1753f4925b9863e28ad4de
/avatars/e56d25a40f2c0e4e6696b9738877fce8.svg
Forrest Iandola
user
forresti
674e953b1ff3268f006ab9f4
Raghuraman Krishnamoorthi
674e953b1ff3268f006ab9f5
Bilge Soran
674e953b1ff3268f006ab9f6
Vikas Chandra
admin_assigned
2024-12-03T16:16:17.253Z
6566681f7b5ed0735812af32
/avatars/424905abb2973954b0850c592743b6fb.svg
Vikas Chandra
user
vchandra
Efficient Track Anything
Segment Anything Model 2 (SAM 2) has emerged as a powerful tool for video
object segmentation and tracking anything. Key components of SAM 2 that drive
the impressive video object segmentation performance include a large multistage
image encoder for frame feature extraction and a memory mechanism that stores
memory contexts from past frames to help current frame segmentation. The high
computation complexity of multistage image encoder and memory module has
limited its applications in real-world tasks, e.g., video object segmentation
on mobile devices. To address this limitation, we propose EfficientTAMs,
lightweight track anything models that produce high-quality results with low
latency and model size. Our idea is based on revisiting the plain,
nonhierarchical Vision Transformer (ViT) as an image encoder for video object
segmentation, and introducing an efficient memory module, which reduces the
complexity for both frame feature extraction and memory computation for current
frame segmentation. We take vanilla lightweight ViTs and efficient memory
module to build EfficientTAMs, and train the models on SA-1B and SA-V datasets
for video object segmentation and track anything tasks. We evaluate on multiple
video segmentation benchmarks including semi-supervised VOS and promptable
video segmentation, and find that our proposed EfficientTAM with vanilla ViT
perform comparably to SAM 2 model (HieraB+SAM 2) with ~2x speedup on A100 and
~2.4x parameter reduction. On segment anything image tasks, our EfficientTAMs
also perform favorably over original SAM with ~20x speedup on A100 and ~20x
parameter reduction. On mobile devices such as iPhone 15 Pro Max, our
EfficientTAMs can run at ~10 FPS for performing video object segmentation with
reasonable quality, highlighting the capability of small models for on-device
video object segmentation applications.
EfficientTAMs employ lightweight Vision Transformers and an efficient memory module for video object segmentation, achieving high performance with reduced complexity and size suitable for mobile devices.
Vision Transformer ViT memory module video object segmentation SA-1B SA-V HieraB+SAM 2 semi-supervised VOS promptable video segmentation EfficientTAMs