title: OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist
    Autonomous Agents for Desktop and Web
summary: For decades, human-computer interaction has fundamentally been manual. Even
  today, almost all productive work done on the computer necessitates human input
  at every step. Autonomous virtual agents represent an exciting step in
  automating many of these menial tasks. Virtual agents would empower users with
  limited technical proficiency to harness the full possibilities of computer
  systems. They could also enable the efficient streamlining of numerous computer
  tasks, ranging from calendar management to complex travel bookings, with
  minimal human intervention. In this paper, we introduce OmniACT, the
  first-of-a-kind dataset and benchmark for assessing an agent's capability to
  generate executable programs to accomplish computer tasks. Our scope extends
  beyond traditional web automation, covering a diverse range of desktop
  applications. The dataset consists of fundamental tasks such as "Play the next
  song", as well as longer horizon tasks such as "Send an email to John Doe
  mentioning the time and place to meet". Specifically, given a pair of screen
  image and a visually-grounded natural language task, the goal is to generate a
  script capable of fully executing the task. We run several strong baseline
  language model agents on our benchmark. The strongest baseline, GPT-4, performs
  the best on our benchmark However, its performance level still reaches only 15%
  of the human proficiency in generating executable scripts capable of completing
  the task, demonstrating the challenge of our task for conventional web agents.
  Our benchmark provides a platform to measure and evaluate the progress of
  language model agents in automating computer tasks and motivates future work
  towards building multimodal models that bridge large language models and the
  visual grounding of computer screens.
paper_authors._id: 65deaf934d56235a39f5a1e7
paper_authors.name: Raghav Kapoor
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-29T08:40:40.489Z
paper_authors._id: 64232ac4eda6dc2bc77f7774
paper_authors.avatarUrl: /avatars/5e088d5a150e47288227d6d30925aee0.svg
paper_authors.fullname: Raghav Kapoor
paper_authors.type: user
paper_authors.user: Raghav1606
paper_authors._id: 65deaf934d56235a39f5a1e8
paper_authors.name: Yash Parag Butala
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-29T11:46:59.850Z
paper_authors._id: 65f5d97d1139c3cd6864ef57
paper_authors.avatarUrl: /avatars/7962eccc3330d30940bff9b50f536532.svg
paper_authors.fullname: Yash Butala
paper_authors.type: user
paper_authors.user: Yash-Butala
paper_authors._id: 65deaf934d56235a39f5a1e9
paper_authors.name: Melisa Russak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T07:47:21.522Z
paper_authors._id: 60e61b3969bd0df25c9375da
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625692968400-noauth.jpeg
paper_authors.fullname: Melisa Russak
paper_authors.type: user
paper_authors.user: melisa
paper_authors._id: 65deaf934d56235a39f5a1ea
paper_authors.name: Jing Yu Koh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:44:20.577Z
paper_authors._id: 63877741189d6915f1d6291e
paper_authors.avatarUrl: /avatars/3110962a08c4613f0f33a4f6d0ff69ce.svg
paper_authors.fullname: Jing Yu Koh
paper_authors.type: user
paper_authors.user: jykoh
paper_authors._id: 65deaf934d56235a39f5a1eb
paper_authors.name: Kiran Kamble
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:44:26.620Z
paper_authors._id: 621d6f532165dc431641e438
paper_authors.avatarUrl: /avatars/56ccef10a8426d7160ef3586a771bd63.svg
paper_authors.fullname: Kiran Kamble
paper_authors.type: user
paper_authors.user: kiranr
paper_authors._id: 65deaf934d56235a39f5a1ec
paper_authors.name: Waseem Alshikh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:44:33.793Z
paper_authors._id: 60cd486d723acf5eb46fe8d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60cd486d723acf5eb46fe8d3/Z1bD1kjvZ0QAOjZna41Xr.jpeg
paper_authors.fullname: Waseem AlShikh
paper_authors.type: user
paper_authors.user: wassemgtk
paper_authors._id: 65deaf934d56235a39f5a1ed
paper_authors.name: Ruslan Salakhutdinov
paper_title: OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist
    Autonomous Agents for Desktop and Web
paper_summary: For decades, human-computer interaction has fundamentally been manual. Even
  today, almost all productive work done on the computer necessitates human input
  at every step. Autonomous virtual agents represent an exciting step in
  automating many of these menial tasks. Virtual agents would empower users with
  limited technical proficiency to harness the full possibilities of computer
  systems. They could also enable the efficient streamlining of numerous computer
  tasks, ranging from calendar management to complex travel bookings, with
  minimal human intervention. In this paper, we introduce OmniACT, the
  first-of-a-kind dataset and benchmark for assessing an agent's capability to
  generate executable programs to accomplish computer tasks. Our scope extends
  beyond traditional web automation, covering a diverse range of desktop
  applications. The dataset consists of fundamental tasks such as "Play the next
  song", as well as longer horizon tasks such as "Send an email to John Doe
  mentioning the time and place to meet". Specifically, given a pair of screen
  image and a visually-grounded natural language task, the goal is to generate a
  script capable of fully executing the task. We run several strong baseline
  language model agents on our benchmark. The strongest baseline, GPT-4, performs
  the best on our benchmark However, its performance level still reaches only 15%
  of the human proficiency in generating executable scripts capable of completing
  the task, demonstrating the challenge of our task for conventional web agents.
  Our benchmark provides a platform to measure and evaluate the progress of
  language model agents in automating computer tasks and motivates future work
  towards building multimodal models that bridge large language models and the
  visual grounding of computer screens.
paper_ai_keywords: OmniACT virtual agents autonomous virtual agents executable programs visually-grounded natural language task script desktop applications multimodal models large language models