title: GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on
    Mobile Devices
summary: Smartphone users often navigate across multiple applications (apps) to
  complete tasks such as sharing content between social media platforms.
  Autonomous Graphical User Interface (GUI) navigation agents can enhance user
  experience in communication, entertainment, and productivity by streamlining
  workflows and reducing manual intervention. However, prior GUI agents often
  trained with datasets comprising simple tasks that can be completed within a
  single app, leading to poor performance in cross-app navigation. To address
  this problem, we introduce GUI Odyssey, a comprehensive dataset for training
  and evaluating cross-app navigation agents. GUI Odyssey consists of 7,735
  episodes from 6 mobile devices, spanning 6 types of cross-app tasks, 201 apps,
  and 1.4K app combos. Leveraging GUI Odyssey, we developed OdysseyAgent, a
  multimodal cross-app navigation agent by fine-tuning the Qwen-VL model with a
  history resampling module. Extensive experiments demonstrate OdysseyAgent's
  superior accuracy compared to existing models. For instance, OdysseyAgent
  surpasses fine-tuned Qwen-VL and zero-shot GPT-4V by 1.44\% and 55.49\%
  in-domain accuracy, and 2.29\% and 48.14\% out-of-domain accuracy on average.
  The dataset and code will be released in
  https://github.com/OpenGVLab/GUI-Odyssey.
paper_authors._id: 666fd4c3cc1f54dfa9132919
paper_authors.name: Quanfeng Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:14:28.922Z
paper_authors._id: 653a483dacdeea08424ef55d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tYG2isIZBLCaBbWABatgK.png
paper_authors.fullname: Quanfeng Lu
paper_authors.type: user
paper_authors.user: hflqf88888
paper_authors._id: 666fd4c3cc1f54dfa913291a
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:14:35.766Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 666fd4c3cc1f54dfa913291b
paper_authors.name: Zitao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:14:41.384Z
paper_authors._id: 666fefc983571a7a055a9a28
paper_authors.avatarUrl: /avatars/a19a0cf6c3315f16f6adc8cbae8238be.svg
paper_authors.fullname: Zitao Liu
paper_authors.type: user
paper_authors.user: windlao
paper_authors._id: 666fd4c3cc1f54dfa913291c
paper_authors.name: Fanqing Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:14:47.622Z
paper_authors._id: 640b37b2bab5ca8fbe7df8f2
paper_authors.avatarUrl: /avatars/c7bef45efad6a0d911a720e2236fcba5.svg
paper_authors.fullname: fanqing meng
paper_authors.type: user
paper_authors.user: FanqingM
paper_authors._id: 666fd4c3cc1f54dfa913291d
paper_authors.name: Boxuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:14:53.006Z
paper_authors._id: 664aefb990135abe9ba9b8c9
paper_authors.avatarUrl: /avatars/79658c23555deadcfb097b6ebccbe4a6.svg
paper_authors.fullname: Boxuan Li
paper_authors.type: user
paper_authors.user: liboxuanhk
paper_authors._id: 666fd4c3cc1f54dfa913291e
paper_authors.name: Botong Chen
paper_authors._id: 666fd4c3cc1f54dfa913291f
paper_authors.name: Siyuan Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-16T07:03:32.762Z
paper_authors._id: 634e4120038b5879133552f5
paper_authors.avatarUrl: /avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
paper_authors.fullname: Siyuan
paper_authors.type: user
paper_authors.user: SiyuanH
paper_authors._id: 666fd4c3cc1f54dfa9132920
paper_authors.name: Kaipeng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:31:41.126Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_authors._id: 666fd4c3cc1f54dfa9132921
paper_authors.name: Yu Qiao
paper_authors._id: 666fd4c3cc1f54dfa9132922
paper_authors.name: Ping Luo
paper_title: GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on
    Mobile Devices
paper_summary: Smartphone users often navigate across multiple applications (apps) to
  complete tasks such as sharing content between social media platforms.
  Autonomous Graphical User Interface (GUI) navigation agents can enhance user
  experience in communication, entertainment, and productivity by streamlining
  workflows and reducing manual intervention. However, prior GUI agents often
  trained with datasets comprising simple tasks that can be completed within a
  single app, leading to poor performance in cross-app navigation. To address
  this problem, we introduce GUI Odyssey, a comprehensive dataset for training
  and evaluating cross-app navigation agents. GUI Odyssey consists of 7,735
  episodes from 6 mobile devices, spanning 6 types of cross-app tasks, 201 apps,
  and 1.4K app combos. Leveraging GUI Odyssey, we developed OdysseyAgent, a
  multimodal cross-app navigation agent by fine-tuning the Qwen-VL model with a
  history resampling module. Extensive experiments demonstrate OdysseyAgent's
  superior accuracy compared to existing models. For instance, OdysseyAgent
  surpasses fine-tuned Qwen-VL and zero-shot GPT-4V by 1.44\% and 55.49\%
  in-domain accuracy, and 2.29\% and 48.14\% out-of-domain accuracy on average.
  The dataset and code will be released in
  https://github.com/OpenGVLab/GUI-Odyssey.
paper_ai_keywords: Autonomous Graphical User Interface (GUI) navigation agents GUI Odyssey Qwen-VL model history resampling module multimodal cross-app navigation agent in-domain accuracy out-of-domain accuracy zero-shot GPT-4V