title: Sharingan: Extract User Action Sequence from Desktop Recordings
summary: Video recordings of user activities, particularly desktop recordings, offer a
  rich source of data for understanding user behaviors and automating processes.
  However, despite advancements in Vision-Language Models (VLMs) and their
  increasing use in video analysis, extracting user actions from desktop
  recordings remains an underexplored area. This paper addresses this gap by
  proposing two novel VLM-based methods for user action extraction: the Direct
  Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and
  the Differential Frame-Based Approach (DiffF), which incorporates explicit
  frame differences detected via computer vision techniques. We evaluate these
  methods using a basic self-curated dataset and an advanced benchmark adapted
  from prior work. Our results show that the DF approach achieves an accuracy of
  70% to 80% in identifying user actions, with the extracted action sequences
  being re-playable though Robotic Process Automation. We find that while VLMs
  show potential, incorporating explicit UI changes can degrade performance,
  making the DF approach more reliable. This work represents the first
  application of VLMs for extracting user action sequences from desktop
  recordings, contributing new methods, benchmarks, and insights for future
  research.
paper_authors._id: 6736fbe36eee0fb7dad3f678
paper_authors.name: Yanting Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:07:00.889Z
paper_authors._id: 670d85fe6ee84c30545ea558
paper_authors.avatarUrl: /avatars/516b00b81603e9a67bd2ba104f375ddb.svg
paper_authors.fullname: Yanting Chen
paper_authors.type: user
paper_authors.user: yantc
paper_authors._id: 6736fbe36eee0fb7dad3f679
paper_authors.name: Yi Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:09:21.884Z
paper_authors._id: 6632323d2a4742700e976553
paper_authors.avatarUrl: /avatars/fb6e017afe268f09e0028135d3b7dc7a.svg
paper_authors.fullname: Yi Ren
paper_authors.type: user
paper_authors.user: yiren0724
paper_authors._id: 6736fbe36eee0fb7dad3f67a
paper_authors.name: Xiaoting Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:09:14.845Z
paper_authors._id: 662b0bc9c709a61df8291c0f
paper_authors.avatarUrl: /avatars/16dd4d945e9fbef5ac889a8087101ded.svg
paper_authors.fullname: Xiaoting Qin
paper_authors.type: user
paper_authors.user: XiaotingQin
paper_authors._id: 6736fbe36eee0fb7dad3f67b
paper_authors.name: Jue Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:09:07.328Z
paper_authors._id: 65f40e43653c231cbaf7d1e4
paper_authors.avatarUrl: /avatars/a42ac5454cbe175f04c3420fce90cad2.svg
paper_authors.fullname: Jue Zhang
paper_authors.type: user
paper_authors.user: JueZhang
paper_authors._id: 6736fbe36eee0fb7dad3f67c
paper_authors.name: Kehong Yuan
paper_authors._id: 6736fbe36eee0fb7dad3f67d
paper_authors.name: Lu Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:08:52.787Z
paper_authors._id: 671f6ae374ec77d5788ad487
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wYkPCbD7kXrZgn426ozXP.png
paper_authors.fullname: Han
paper_authors.type: user
paper_authors.user: luhan-microsoft
paper_authors._id: 6736fbe36eee0fb7dad3f67e
paper_authors.name: Qingwei Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:08:34.679Z
paper_authors._id: 652fc9f39bc50a6c0e435224
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652fc9f39bc50a6c0e435224/70OBVDHHBsxG2giJ-E3_1.jpeg
paper_authors.fullname: Lin Qingwei
paper_authors.type: user
paper_authors.user: Eliblo1969
paper_authors._id: 6736fbe36eee0fb7dad3f67f
paper_authors.name: Dongmei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:08:22.520Z
paper_authors._id: 66473d2c7abe6ad66e81a3dd
paper_authors.avatarUrl: /avatars/82f40244806c06ffeaa1c4265e9725ea.svg
paper_authors.fullname: ZHANGDONGMEI
paper_authors.type: user
paper_authors.user: ZDM6426
paper_authors._id: 6736fbe36eee0fb7dad3f680
paper_authors.name: Saravan Rajmohan
paper_authors._id: 6736fbe36eee0fb7dad3f681
paper_authors.name: Qi Zhang
paper_title: Sharingan: Extract User Action Sequence from Desktop Recordings
paper_summary: Video recordings of user activities, particularly desktop recordings, offer a
  rich source of data for understanding user behaviors and automating processes.
  However, despite advancements in Vision-Language Models (VLMs) and their
  increasing use in video analysis, extracting user actions from desktop
  recordings remains an underexplored area. This paper addresses this gap by
  proposing two novel VLM-based methods for user action extraction: the Direct
  Frame-Based Approach (DF), which inputs sampled frames directly into VLMs, and
  the Differential Frame-Based Approach (DiffF), which incorporates explicit
  frame differences detected via computer vision techniques. We evaluate these
  methods using a basic self-curated dataset and an advanced benchmark adapted
  from prior work. Our results show that the DF approach achieves an accuracy of
  70% to 80% in identifying user actions, with the extracted action sequences
  being re-playable though Robotic Process Automation. We find that while VLMs
  show potential, incorporating explicit UI changes can degrade performance,
  making the DF approach more reliable. This work represents the first
  application of VLMs for extracting user action sequences from desktop
  recordings, contributing new methods, benchmarks, and insights for future
  research.
paper_ai_summary: VLM-based methods, Direct Frame-Based Approach and Differential Frame-Based Approach, for extracting user actions from desktop recordings are proposed and evaluated.
paper_ai_keywords: Vision-Language Models (VLMs) Direct Frame-Based Approach Differential Frame-Based Approach computer vision techniques self-curated dataset advanced benchmark action sequences Robotic Process Automation