title: GazeGen: Gaze-Driven User Interaction for Visual Content Generation
summary: We present GazeGen, a user interaction system that generates visual content
  (images and videos) for locations indicated by the user's eye gaze. GazeGen
  allows intuitive manipulation of visual content by targeting regions of
  interest with gaze. Using advanced techniques in object detection and
  generative AI, GazeGen performs gaze-controlled image adding/deleting,
  repositioning, and surface material changes of image objects, and converts
  static images into videos. Central to GazeGen is the DFT Gaze (Distilled and
  Fine-Tuned Gaze) agent, an ultra-lightweight model with only 281K parameters,
  performing accurate real-time gaze predictions tailored to individual users'
  eyes on small edge devices. GazeGen is the first system to combine visual
  content generation with real-time gaze estimation, made possible exclusively by
  DFT Gaze. This real-time gaze estimation enables various visual content
  generation tasks, all controlled by the user's gaze. The input for DFT Gaze is
  the user's eye images, while the inputs for visual content generation are the
  user's view and the predicted gaze point from DFT Gaze. To achieve efficient
  gaze predictions, we derive the small model from a large model (10x larger) via
  novel knowledge distillation and personal adaptation techniques. We integrate
  knowledge distillation with a masked autoencoder, developing a compact yet
  powerful gaze estimation model. This model is further fine-tuned with Adapters,
  enabling highly accurate and personalized gaze predictions with minimal user
  input. DFT Gaze ensures low-latency and precise gaze tracking, supporting a
  wide range of gaze-driven tasks. We validate the performance of DFT Gaze on AEA
  and OpenEDS2020 benchmarks, demonstrating low angular gaze error and low
  latency on the edge device (Raspberry Pi 4). Furthermore, we describe
  applications of GazeGen, illustrating its versatility and effectiveness in
  various usage scenarios.
paper_authors._id: 672d8d9316766a76a7bbbd49
paper_authors.name: He-Yen Hsieh
paper_authors._id: 672d8d9316766a76a7bbbd4a
paper_authors.name: Ziyun Li
paper_authors._id: 672d8d9316766a76a7bbbd4b
paper_authors.name: Sai Qian Zhang
paper_authors._id: 672d8d9316766a76a7bbbd4c
paper_authors.name: Wei-Te Mark Ting
paper_authors._id: 672d8d9316766a76a7bbbd4d
paper_authors.name: Kao-Den Chang
paper_authors._id: 672d8d9316766a76a7bbbd4e
paper_authors.name: Barbara De Salvo
paper_authors._id: 672d8d9316766a76a7bbbd4f
paper_authors.name: Chiao Liu
paper_authors._id: 672d8d9316766a76a7bbbd50
paper_authors.name: H. T. Kung
paper_title: GazeGen: Gaze-Driven User Interaction for Visual Content Generation
paper_summary: We present GazeGen, a user interaction system that generates visual content
  (images and videos) for locations indicated by the user's eye gaze. GazeGen
  allows intuitive manipulation of visual content by targeting regions of
  interest with gaze. Using advanced techniques in object detection and
  generative AI, GazeGen performs gaze-controlled image adding/deleting,
  repositioning, and surface material changes of image objects, and converts
  static images into videos. Central to GazeGen is the DFT Gaze (Distilled and
  Fine-Tuned Gaze) agent, an ultra-lightweight model with only 281K parameters,
  performing accurate real-time gaze predictions tailored to individual users'
  eyes on small edge devices. GazeGen is the first system to combine visual
  content generation with real-time gaze estimation, made possible exclusively by
  DFT Gaze. This real-time gaze estimation enables various visual content
  generation tasks, all controlled by the user's gaze. The input for DFT Gaze is
  the user's eye images, while the inputs for visual content generation are the
  user's view and the predicted gaze point from DFT Gaze. To achieve efficient
  gaze predictions, we derive the small model from a large model (10x larger) via
  novel knowledge distillation and personal adaptation techniques. We integrate
  knowledge distillation with a masked autoencoder, developing a compact yet
  powerful gaze estimation model. This model is further fine-tuned with Adapters,
  enabling highly accurate and personalized gaze predictions with minimal user
  input. DFT Gaze ensures low-latency and precise gaze tracking, supporting a
  wide range of gaze-driven tasks. We validate the performance of DFT Gaze on AEA
  and OpenEDS2020 benchmarks, demonstrating low angular gaze error and low
  latency on the edge device (Raspberry Pi 4). Furthermore, we describe
  applications of GazeGen, illustrating its versatility and effectiveness in
  various usage scenarios.
paper_ai_keywords: object detection generative AI gaze-controlled image adding/deleting repositioning surface material changes DFT Gaze (Distilled and Fine-Tuned Gaze) ultra-lightweight model real-time gaze predictions knowledge distillation masked autoencoder Adapters low-latency gaze tracking AEA benchmark OpenEDS2020 benchmark low angular gaze error