title: The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer
    Use
summary: The recently released model, Claude 3.5 Computer Use, stands out as the first
  frontier AI model to offer computer use in public beta as a graphical user
  interface (GUI) agent. As an early beta, its capability in the real-world
  complex environment remains unknown. In this case study to explore Claude 3.5
  Computer Use, we curate and organize a collection of carefully designed tasks
  spanning a variety of domains and software. Observations from these cases
  demonstrate Claude 3.5 Computer Use's unprecedented ability in end-to-end
  language to desktop actions. Along with this study, we provide an
  out-of-the-box agent framework for deploying API-based GUI automation models
  with easy implementation. Our case studies aim to showcase a groundwork of
  capabilities and limitations of Claude 3.5 Computer Use with detailed analyses
  and bring to the fore questions about planning, action, and critic, which must
  be considered for future improvement. We hope this preliminary exploration will
  inspire future research into the GUI agent community. All the test cases in the
  paper can be tried through the project:
  https://github.com/showlab/computer_use_ootb.
paper_authors._id: 673acfbc4442393f72d541f0
paper_authors.name: Siyuan Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T13:01:57.944Z
paper_authors._id: 6357c9f400f138b8ca551704
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6357c9f400f138b8ca551704/7cUf2RznjAfXNkO_wJ_jl.png
paper_authors.fullname: Siyuan Hu
paper_authors.type: user
paper_authors.user: h-siyuan
paper_authors._id: 673acfbc4442393f72d541f1
paper_authors.name: Mingyu Ouyang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-18T15:19:06.438Z
paper_authors._id: 634e2217c1ce28f1de921708
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634e2217c1ce28f1de921708/XTMB6alYUM0KAUptM98kP.jpeg
paper_authors.fullname: Yang
paper_authors.type: user
paper_authors.user: yyyang
paper_authors._id: 673acfbc4442393f72d541f2
paper_authors.name: Difei Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-18T07:51:04.868Z
paper_authors._id: 661abfba70ec567e154fbb00
paper_authors.avatarUrl: /avatars/46c004650ba15525ccf02d40c9314085.svg
paper_authors.fullname: Difei Gao
paper_authors.type: user
paper_authors.user: QuStar
paper_authors._id: 673acfbc4442393f72d541f3
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-18T07:51:10.297Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer
    Use
paper_summary: The recently released model, Claude 3.5 Computer Use, stands out as the first
  frontier AI model to offer computer use in public beta as a graphical user
  interface (GUI) agent. As an early beta, its capability in the real-world
  complex environment remains unknown. In this case study to explore Claude 3.5
  Computer Use, we curate and organize a collection of carefully designed tasks
  spanning a variety of domains and software. Observations from these cases
  demonstrate Claude 3.5 Computer Use's unprecedented ability in end-to-end
  language to desktop actions. Along with this study, we provide an
  out-of-the-box agent framework for deploying API-based GUI automation models
  with easy implementation. Our case studies aim to showcase a groundwork of
  capabilities and limitations of Claude 3.5 Computer Use with detailed analyses
  and bring to the fore questions about planning, action, and critic, which must
  be considered for future improvement. We hope this preliminary exploration will
  inspire future research into the GUI agent community. All the test cases in the
  paper can be tried through the project:
  https://github.com/showlab/computer_use_ootb.
paper_ai_keywords: graphical user interface (GUI) API-based GUI automation end-to-end language to desktop-actions out-of-the-box agent framework maximum likelihood training