title: Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
summary: We address the problem of gaze target estimation, which aims to predict where
  a person is looking in a scene. Predicting a person's gaze target requires
  reasoning both about the person's appearance and the contents of the scene.
  Prior works have developed increasingly complex, hand-crafted pipelines for
  gaze target estimation that carefully fuse features from separate scene
  encoders, head encoders, and auxiliary models for signals like depth and pose.
  Motivated by the success of general-purpose feature extractors on a variety of
  visual tasks, we propose Gaze-LLE, a novel transformer framework that
  streamlines gaze target estimation by leveraging features from a frozen DINOv2
  encoder. We extract a single feature representation for the scene, and apply a
  person-specific positional prompt to decode gaze with a lightweight module. We
  demonstrate state-of-the-art performance across several gaze benchmarks and
  provide extensive analysis to validate our design choices. Our code is
  available at: http://github.com/fkryan/gazelle .
paper_authors._id: 675ba24cc7ceaec7e65066d0
paper_authors.name: Fiona Ryan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:25.678Z
paper_authors._id: 656a4b89e8bf55919a313920
paper_authors.avatarUrl: /avatars/c903ec6e1c7952c5a200edd9f8cfcee7.svg
paper_authors.fullname: Fiona Ryan
paper_authors.type: user
paper_authors.user: fkryan
paper_authors._id: 675ba24cc7ceaec7e65066d1
paper_authors.name: Ajay Bati
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:58:25.614Z
paper_authors._id: 638ed41149de7ae552de38cf
paper_authors.avatarUrl: /avatars/eb0af76e6557b7ec26893902b223254d.svg
paper_authors.fullname: Ajay Bati
paper_authors.type: user
paper_authors.user: ajbot
paper_authors._id: 675ba24cc7ceaec7e65066d2
paper_authors.name: Sangmin Lee
paper_authors._id: 675ba24cc7ceaec7e65066d3
paper_authors.name: Daniel Bolya
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:58:54.549Z
paper_authors._id: 62ff05ab9bfda579504d81b2
paper_authors.avatarUrl: /avatars/94590ed960b6df3fc1ababa31cd13f17.svg
paper_authors.fullname: Daniel Bolya
paper_authors.type: user
paper_authors.user: dbolya
paper_authors._id: 675ba24cc7ceaec7e65066d4
paper_authors.name: Judy Hoffman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:59:00.557Z
paper_authors._id: 6477790c4ae93470ffc297bb
paper_authors.avatarUrl: /avatars/f8d5e08719297a335731a117416d6336.svg
paper_authors.fullname: Judy Hoffman
paper_authors.type: user
paper_authors.user: judyhoffman
paper_authors._id: 675ba24cc7ceaec7e65066d5
paper_authors.name: James M. Rehg
paper_title: Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
paper_summary: We address the problem of gaze target estimation, which aims to predict where
  a person is looking in a scene. Predicting a person's gaze target requires
  reasoning both about the person's appearance and the contents of the scene.
  Prior works have developed increasingly complex, hand-crafted pipelines for
  gaze target estimation that carefully fuse features from separate scene
  encoders, head encoders, and auxiliary models for signals like depth and pose.
  Motivated by the success of general-purpose feature extractors on a variety of
  visual tasks, we propose Gaze-LLE, a novel transformer framework that
  streamlines gaze target estimation by leveraging features from a frozen DINOv2
  encoder. We extract a single feature representation for the scene, and apply a
  person-specific positional prompt to decode gaze with a lightweight module. We
  demonstrate state-of-the-art performance across several gaze benchmarks and
  provide extensive analysis to validate our design choices. Our code is
  available at: http://github.com/fkryan/gazelle .
paper_ai_summary: Gaze-LLE, a transformer framework using features from a DINOv2 encoder, achieves state-of-the-art performance in gaze target estimation by integrating scene and person-specific information.
paper_ai_keywords: gaze target estimation transformer framework DINOv2 encoder feature representation positional prompt scene encoders head encoders auxiliary models gaze benchmarks