OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist
  Autonomous Agents for Desktop and Web
For decades, human-computer interaction has fundamentally been manual. Even
today, almost all productive work done on the computer necessitates human input
at every step. Autonomous virtual agents represent an exciting step in
automating many of these menial tasks. Virtual agents would empower users with
limited technical proficiency to harness the full possibilities of computer
systems. They could also enable the efficient streamlining of numerous computer
tasks, ranging from calendar management to complex travel bookings, with
minimal human intervention. In this paper, we introduce OmniACT, the
first-of-a-kind dataset and benchmark for assessing an agent's capability to
generate executable programs to accomplish computer tasks. Our scope extends
beyond traditional web automation, covering a diverse range of desktop
applications. The dataset consists of fundamental tasks such as "Play the next
song", as well as longer horizon tasks such as "Send an email to John Doe
mentioning the time and place to meet". Specifically, given a pair of screen
image and a visually-grounded natural language task, the goal is to generate a
script capable of fully executing the task. We run several strong baseline
language model agents on our benchmark. The strongest baseline, GPT-4, performs
the best on our benchmark However, its performance level still reaches only 15%
of the human proficiency in generating executable scripts capable of completing
the task, demonstrating the challenge of our task for conventional web agents.
Our benchmark provides a platform to measure and evaluate the progress of
language model agents in automating computer tasks and motivates future work
towards building multimodal models that bridge large language models and the
visual grounding of computer screens.
65deaf934d56235a39f5a1e7
Raghav Kapoor
claimed_verified
2024-02-29T08:40:40.489Z
64232ac4eda6dc2bc77f7774
/avatars/5e088d5a150e47288227d6d30925aee0.svg
Raghav Kapoor
user
Raghav1606
65deaf934d56235a39f5a1e8
Yash Parag Butala
claimed_verified
2024-04-29T11:46:59.850Z
65f5d97d1139c3cd6864ef57
/avatars/7962eccc3330d30940bff9b50f536532.svg
Yash Butala
user
Yash-Butala
65deaf934d56235a39f5a1e9
Melisa Russak
admin_assigned
2024-08-28T07:47:21.522Z
60e61b3969bd0df25c9375da
https://cdn-avatars.huggingface.co/v1/production/uploads/1625692968400-noauth.jpeg
Melisa Russak
user
melisa
65deaf934d56235a39f5a1ea
Jing Yu Koh
admin_assigned
2024-02-28T12:44:20.577Z
63877741189d6915f1d6291e
/avatars/3110962a08c4613f0f33a4f6d0ff69ce.svg
Jing Yu Koh
user
jykoh
65deaf934d56235a39f5a1eb
Kiran Kamble
admin_assigned
2024-02-28T12:44:26.620Z
621d6f532165dc431641e438
/avatars/56ccef10a8426d7160ef3586a771bd63.svg
Kiran Kamble
user
kiranr
65deaf934d56235a39f5a1ec
Waseem Alshikh
admin_assigned
2024-02-28T12:44:33.793Z
60cd486d723acf5eb46fe8d3
https://cdn-avatars.huggingface.co/v1/production/uploads/60cd486d723acf5eb46fe8d3/Z1bD1kjvZ0QAOjZna41Xr.jpeg
Waseem AlShikh
user
wassemgtk
65deaf934d56235a39f5a1ed
Ruslan Salakhutdinov
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist
  Autonomous Agents for Desktop and Web
For decades, human-computer interaction has fundamentally been manual. Even
today, almost all productive work done on the computer necessitates human input
at every step. Autonomous virtual agents represent an exciting step in
automating many of these menial tasks. Virtual agents would empower users with
limited technical proficiency to harness the full possibilities of computer
systems. They could also enable the efficient streamlining of numerous computer
tasks, ranging from calendar management to complex travel bookings, with
minimal human intervention. In this paper, we introduce OmniACT, the
first-of-a-kind dataset and benchmark for assessing an agent's capability to
generate executable programs to accomplish computer tasks. Our scope extends
beyond traditional web automation, covering a diverse range of desktop
applications. The dataset consists of fundamental tasks such as "Play the next
song", as well as longer horizon tasks such as "Send an email to John Doe
mentioning the time and place to meet". Specifically, given a pair of screen
image and a visually-grounded natural language task, the goal is to generate a
script capable of fully executing the task. We run several strong baseline
language model agents on our benchmark. The strongest baseline, GPT-4, performs
the best on our benchmark However, its performance level still reaches only 15%
of the human proficiency in generating executable scripts capable of completing
the task, demonstrating the challenge of our task for conventional web agents.
Our benchmark provides a platform to measure and evaluate the progress of
language model agents in automating computer tasks and motivates future work
towards building multimodal models that bridge large language models and the
visual grounding of computer screens.
A dataset named OmniACT is introduced to evaluate agents' capability to generate executable scripts for various computer tasks, highlighting the challenge for current models in achieving human-level proficiency.
autonomous virtual agents OmniACT dataset benchmark executable programs desktop applications screen image visually-grounded natural language language model agents GPT-4 multimodal models