title: Follow Anything: Open-set detection, tracking, and following in
    real-time
summary: Tracking and following objects of interest is critical to several robotics
  use cases, ranging from industrial automation to logistics and warehousing, to
  healthcare and security. In this paper, we present a robotic system to detect,
  track, and follow any object in real-time. Our approach, dubbed ``follow
  anything'' (FAn), is an open-vocabulary and multimodal model -- it is not
  restricted to concepts seen at training time and can be applied to novel
  classes at inference time using text, images, or click queries. Leveraging rich
  visual descriptors from large-scale pre-trained models (foundation models), FAn
  can detect and segment objects by matching multimodal queries (text, images,
  clicks) against an input image sequence. These detected and segmented objects
  are tracked across image frames, all while accounting for occlusion and object
  re-emergence. We demonstrate FAn on a real-world robotic system (a micro aerial
  vehicle) and report its ability to seamlessly follow the objects of interest in
  a real-time control loop. FAn can be deployed on a laptop with a lightweight
  (6-8 GB) graphics card, achieving a throughput of 6-20 frames per second. To
  enable rapid adoption, deployment, and extensibility, we open-source all our
  code on our project webpage at https://github.com/alaamaalouf/FollowAnything .
  We also encourage the reader the watch our 5-minutes explainer video in this
  https://www.youtube.com/watch?v=6Mgt3EPytrw .
paper_authors._id: 64d5af0595cf13a3818163dd
paper_authors.name: Alaa Maalouf
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-13T00:11:26.704Z
paper_authors._id: 64d65549d7e30889c6697802
paper_authors.avatarUrl: /avatars/f45d58fb1428acd06af19fd701407cef.svg
paper_authors.fullname: Alaa Maalouf
paper_authors.type: user
paper_authors.user: alaamaalouf
paper_authors._id: 64d5af0595cf13a3818163de
paper_authors.name: Ninad Jadhav
paper_authors._id: 64d5af0595cf13a3818163df
paper_authors.name: Krishna Murthy Jatavallabhula
paper_authors._id: 64d5af0595cf13a3818163e0
paper_authors.name: Makram Chahine
paper_authors._id: 64d5af0595cf13a3818163e1
paper_authors.name: Daniel M. Vogt
paper_authors._id: 64d5af0595cf13a3818163e2
paper_authors.name: Robert J. Wood
paper_authors._id: 64d5af0595cf13a3818163e3
paper_authors.name: Antonio Torralba
paper_authors._id: 64d5af0595cf13a3818163e4
paper_authors.name: Daniela Rus
paper_title: Follow Anything: Open-set detection, tracking, and following in
    real-time
paper_summary: Tracking and following objects of interest is critical to several robotics
  use cases, ranging from industrial automation to logistics and warehousing, to
  healthcare and security. In this paper, we present a robotic system to detect,
  track, and follow any object in real-time. Our approach, dubbed ``follow
  anything'' (FAn), is an open-vocabulary and multimodal model -- it is not
  restricted to concepts seen at training time and can be applied to novel
  classes at inference time using text, images, or click queries. Leveraging rich
  visual descriptors from large-scale pre-trained models (foundation models), FAn
  can detect and segment objects by matching multimodal queries (text, images,
  clicks) against an input image sequence. These detected and segmented objects
  are tracked across image frames, all while accounting for occlusion and object
  re-emergence. We demonstrate FAn on a real-world robotic system (a micro aerial
  vehicle) and report its ability to seamlessly follow the objects of interest in
  a real-time control loop. FAn can be deployed on a laptop with a lightweight
  (6-8 GB) graphics card, achieving a throughput of 6-20 frames per second. To
  enable rapid adoption, deployment, and extensibility, we open-source all our
  code on our project webpage at https://github.com/alaamaalouf/FollowAnything .
  We also encourage the reader the watch our 5-minutes explainer video in this
  https://www.youtube.com/watch?v=6Mgt3EPytrw .
paper_ai_summary: FAn, an open-vocabulary and multimodal robotic system, detects, tracks, and follows objects in real-time using pre-trained models and multimodal queries, and is deployable on a laptop with a lightweight GPU.
paper_ai_keywords: follow anything open-vocabulary multimodal model pre-trained models visual descriptors multimodal queries object detection object segmentation object tracking occlusion object re-emergence micro aerial vehicle real-time control loop