Answers.Body: <p>There's a couple:</p>
  <p>Mind2Web: <a href="https://arxiv.org/pdf/2306.06070.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2306.06070.pdf</a></p>
  <p>Webshop: <a href="https://webshop-pnlp.github.io/" rel="nofollow noreferrer">https://webshop-pnlp.github.io/</a></p>
  <p>MiniWoB++: <a href="https://arxiv.org/pdf/2305.11854.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2305.11854.pdf</a></p>
  <p>These papers also reference other, similar datasets.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-16T05:44:48.967Z
Answers.LastActivityDate: 2023-07-16T05:45:25.117Z
Answers.LastEditDate: 2023-07-16T05:45:25.117Z
Answers.UserDisplayName: Alexander Wan
Body: <p>Here's a demo for <a href="https://www.adept.ai/blog/act-1" rel="nofollow noreferrer">Adept ACT-1 for Transformers</a>. I don't doubt that one could create a demo video using zero-shot; actually I tested just now and the basic chat.openai.com interface was able to do some web browsing stuff on the first try if I just prompted it with a list of API methods (e.g. <code>el.click</code>) and streamed the DOM to it.</p>
  <p>But I assume that, to take their product to the next level, they must be training or fine-tuning on some actual multimodal data, e.g. session recordings labeled with natural language? Am I wrong?</p>
  <p>Are there any open-access datasets like this that could be used for open-source models? I'm actually interested in native (e.g. pixel + natural language to game input) more than browser.</p>
  
Tags: <large-language-models><training-datasets>
Title: Open access Adept-like dataset? (LLM-to-computer-input)
UserDisplayName: llllvvuu
ThreadText: # Open access Adept-like dataset? (LLM-to-computer-input)
  
  (Asked by: username_0 on 2023-07-15)
  
  Here's a demo for [Adept ACT-1 for Transformers](https://www.adept.ai/blog/act-1). I don't doubt that one could create a demo video using zero-shot; actually I tested just now and the basic chat.openai.com interface was able to do some web browsing stuff on the first try if I just prompted it with a list of API methods (e.g. `el.click`) and streamed the DOM to it.
  
  But I assume that, to take their product to the next level, they must be training or fine-tuning on some actual multimodal data, e.g. session recordings labeled with natural language? Am I wrong?
  
  Are there any open-access datasets like this that could be used for open-source models? I'm actually interested in native (e.g. pixel + natural language to game input) more than browser.
  
  **Tags:** large-language-models, training-datasets
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-16. Score: 0 | Accepted answer
  
  There's a couple:
  
  Mind2Web: <https://arxiv.org/pdf/2306.06070.pdf>
  
  Webshop: <https://webshop-pnlp.github.io/>
  
  MiniWoB++: <https://arxiv.org/pdf/2305.11854.pdf>
  
  These papers also reference other, similar datasets.
  
  
  ---
  
  