title: SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking
    with Motion-Aware Memory
summary: The Segment Anything Model 2 (SAM 2) has demonstrated strong performance in
  object segmentation tasks but faces challenges in visual object tracking,
  particularly when managing crowded scenes with fast-moving or self-occluding
  objects. Furthermore, the fixed-window memory approach in the original model
  does not consider the quality of memories selected to condition the image
  features for the next frame, leading to error propagation in videos. This paper
  introduces SAMURAI, an enhanced adaptation of SAM 2 specifically designed for
  visual object tracking. By incorporating temporal motion cues with the proposed
  motion-aware memory selection mechanism, SAMURAI effectively predicts object
  motion and refines mask selection, achieving robust, accurate tracking without
  the need for retraining or fine-tuning. SAMURAI operates in real-time and
  demonstrates strong zero-shot performance across diverse benchmark datasets,
  showcasing its ability to generalize without fine-tuning. In evaluations,
  SAMURAI achieves significant improvements in success rate and precision over
  existing trackers, with a 7.1% AUC gain on LaSOT_{ext} and a 3.5% AO
  gain on GOT-10k. Moreover, it achieves competitive results compared to fully
  supervised methods on LaSOT, underscoring its robustness in complex tracking
  scenarios and its potential for real-world applications in dynamic
  environments. Code and results are available at
  https://github.com/yangchris11/samurai.
paper_authors._id: 673d668e7c9e8932533502b8
paper_authors.name: Cheng-Yen Yang
paper_authors._id: 673d668e7c9e8932533502b9
paper_authors.name: Hsiang-Wei Huang
paper_authors._id: 673d668e7c9e8932533502ba
paper_authors.name: Wenhao Chai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-20T10:08:27.233Z
paper_authors._id: 637c7503fe115289cfecbe6b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676361945047-637c7503fe115289cfecbe6b.jpeg
paper_authors.fullname: Wenhao Chai
paper_authors.type: user
paper_authors.user: wchai
paper_authors._id: 673d668e7c9e8932533502bb
paper_authors.name: Zhongyu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T08:09:49.008Z
paper_authors._id: 65714e2a6bbfca64615f9931
paper_authors.avatarUrl: /avatars/9048b08655a7b0a877a47c7424abe50e.svg
paper_authors.fullname: Zhongyu Jiang
paper_authors.type: user
paper_authors.user: Nitre
paper_authors._id: 673d668e7c9e8932533502bc
paper_authors.name: Jenq-Neng Hwang
paper_title: SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking
    with Motion-Aware Memory
paper_summary: The Segment Anything Model 2 (SAM 2) has demonstrated strong performance in
  object segmentation tasks but faces challenges in visual object tracking,
  particularly when managing crowded scenes with fast-moving or self-occluding
  objects. Furthermore, the fixed-window memory approach in the original model
  does not consider the quality of memories selected to condition the image
  features for the next frame, leading to error propagation in videos. This paper
  introduces SAMURAI, an enhanced adaptation of SAM 2 specifically designed for
  visual object tracking. By incorporating temporal motion cues with the proposed
  motion-aware memory selection mechanism, SAMURAI effectively predicts object
  motion and refines mask selection, achieving robust, accurate tracking without
  the need for retraining or fine-tuning. SAMURAI operates in real-time and
  demonstrates strong zero-shot performance across diverse benchmark datasets,
  showcasing its ability to generalize without fine-tuning. In evaluations,
  SAMURAI achieves significant improvements in success rate and precision over
  existing trackers, with a 7.1% AUC gain on LaSOT_{ext} and a 3.5% AO
  gain on GOT-10k. Moreover, it achieves competitive results compared to fully
  supervised methods on LaSOT, underscoring its robustness in complex tracking
  scenarios and its potential for real-world applications in dynamic
  environments. Code and results are available at
  https://github.com/yangchris11/samurai.
paper_ai_keywords: object segmentation visual object tracking crowded scenes self-occluding objects fixed-window memory motion-aware memory selection mechanism object motion prediction mask selection zero-shot performance benchmark datasets success rate precision AUC AO fully supervised methods dynamic environments