title: Large Action Models: From Inception to Implementation
summary: As AI continues to advance, there is a growing demand for systems that go
  beyond language-based assistance and move toward intelligent agents capable of
  performing real-world actions. This evolution requires the transition from
  traditional Large Language Models (LLMs), which excel at generating textual
  responses, to Large Action Models (LAMs), designed for action generation and
  execution within dynamic environments. Enabled by agent systems, LAMs hold the
  potential to transform AI from passive language understanding to active task
  completion, marking a significant milestone in the progression toward
  artificial general intelligence.
    In this paper, we present a comprehensive framework for developing LAMs,
  offering a systematic approach to their creation, from inception to deployment.
  We begin with an overview of LAMs, highlighting their unique characteristics
  and delineating their differences from LLMs. Using a Windows OS-based agent as
  a case study, we provide a detailed, step-by-step guide on the key stages of
  LAM development, including data collection, model training, environment
  integration, grounding, and evaluation. This generalizable workflow can serve
  as a blueprint for creating functional LAMs in various application domains. We
  conclude by identifying the current limitations of LAMs and discussing
  directions for future research and industrial deployment, emphasizing the
  challenges and opportunities that lie ahead in realizing the full potential of
  LAMs in real-world applications.
    The code for the data collection process utilized in this paper is publicly
  available at: https://github.com/microsoft/UFO/tree/main/dataflow, and
  comprehensive documentation can be found at
  https://microsoft.github.io/UFO/dataflow/overview/.
paper_authors._id: 675f9dce5dca7e3c5c1f0950
paper_authors.name: Lu Wang
paper_authors._id: 675f9dce5dca7e3c5c1f0951
paper_authors.name: Fangkai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T09:18:20.886Z
paper_authors._id: 669dcf6200970c3b27aafa5d
paper_authors.avatarUrl: /avatars/bb9ed5ff86326fdaeb184c6b0e40f74f.svg
paper_authors.fullname: kaikai yang
paper_authors.type: user
paper_authors.user: keanudicap
paper_authors._id: 675f9dce5dca7e3c5c1f0952
paper_authors.name: Chaoyun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:13:55.597Z
paper_authors._id: 654dbac9938fbf1e696be8aa
paper_authors.avatarUrl: /avatars/b3c4035c48169c1bfb04a439fce3499f.svg
paper_authors.fullname: Chaoyun Zhang
paper_authors.type: user
paper_authors.user: vyokky
paper_authors._id: 675f9dce5dca7e3c5c1f0953
paper_authors.name: Junting Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:14:25.301Z
paper_authors._id: 6724f7b2f0e98abd46cb94d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/O47r1efbW0MYxGyhixw9k.png
paper_authors.fullname: lu junior ting
paper_authors.type: user
paper_authors.user: Lujunting
paper_authors._id: 675f9dce5dca7e3c5c1f0954
paper_authors.name: Jiaxu Qian
paper_authors._id: 675f9dce5dca7e3c5c1f0955
paper_authors.name: Shilin He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:14:11.939Z
paper_authors._id: 62c6df026a092eda1f1ab6e5
paper_authors.avatarUrl: /avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
paper_authors.fullname: Shilin He
paper_authors.type: user
paper_authors.user: shilhe
paper_authors._id: 675f9dce5dca7e3c5c1f0956
paper_authors.name: Pu Zhao
paper_authors._id: 675f9dce5dca7e3c5c1f0957
paper_authors.name: Bo Qiao
paper_authors._id: 675f9dce5dca7e3c5c1f0958
paper_authors.name: Ray Huang
paper_authors._id: 675f9dce5dca7e3c5c1f0959
paper_authors.name: Si Qin
paper_authors._id: 675f9dce5dca7e3c5c1f095a
paper_authors.name: Qisheng Su
paper_authors._id: 675f9dce5dca7e3c5c1f095b
paper_authors.name: Jiayi Ye
paper_authors._id: 675f9dce5dca7e3c5c1f095c
paper_authors.name: Yudi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:14.067Z
paper_authors._id: 662a5511be52f6c4d5dd0b64
paper_authors.avatarUrl: /avatars/759686e916c36ebea5908e400d1cf25b.svg
paper_authors.fullname: Yudi Zhang
paper_authors.type: user
paper_authors.user: ReedZyd
paper_authors._id: 675f9dce5dca7e3c5c1f095d
paper_authors.name: Jian-Guang Lou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:11:51.046Z
paper_authors._id: 6554338ba4de813482e91ad5
paper_authors.avatarUrl: /avatars/bee879fa4a818fd6c070ee001769ce05.svg
paper_authors.fullname: Jian-Guang Lou
paper_authors.type: user
paper_authors.user: substill
paper_authors._id: 675f9dce5dca7e3c5c1f095e
paper_authors.name: Qingwei Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:11:45.046Z
paper_authors._id: 652fc9f39bc50a6c0e435224
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652fc9f39bc50a6c0e435224/70OBVDHHBsxG2giJ-E3_1.jpeg
paper_authors.fullname: Lin Qingwei
paper_authors.type: user
paper_authors.user: Eliblo1969
paper_authors._id: 675f9dce5dca7e3c5c1f095f
paper_authors.name: Saravan Rajmohan
paper_authors._id: 675f9dce5dca7e3c5c1f0960
paper_authors.name: Dongmei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:11:13.362Z
paper_authors._id: 66473d2c7abe6ad66e81a3dd
paper_authors.avatarUrl: /avatars/82f40244806c06ffeaa1c4265e9725ea.svg
paper_authors.fullname: ZHANGDONGMEI
paper_authors.type: user
paper_authors.user: ZDM6426
paper_authors._id: 675f9dce5dca7e3c5c1f0961
paper_authors.name: Qi Zhang
paper_title: Large Action Models: From Inception to Implementation
paper_summary: As AI continues to advance, there is a growing demand for systems that go
  beyond language-based assistance and move toward intelligent agents capable of
  performing real-world actions. This evolution requires the transition from
  traditional Large Language Models (LLMs), which excel at generating textual
  responses, to Large Action Models (LAMs), designed for action generation and
  execution within dynamic environments. Enabled by agent systems, LAMs hold the
  potential to transform AI from passive language understanding to active task
  completion, marking a significant milestone in the progression toward
  artificial general intelligence.
    In this paper, we present a comprehensive framework for developing LAMs,
  offering a systematic approach to their creation, from inception to deployment.
  We begin with an overview of LAMs, highlighting their unique characteristics
  and delineating their differences from LLMs. Using a Windows OS-based agent as
  a case study, we provide a detailed, step-by-step guide on the key stages of
  LAM development, including data collection, model training, environment
  integration, grounding, and evaluation. This generalizable workflow can serve
  as a blueprint for creating functional LAMs in various application domains. We
  conclude by identifying the current limitations of LAMs and discussing
  directions for future research and industrial deployment, emphasizing the
  challenges and opportunities that lie ahead in realizing the full potential of
  LAMs in real-world applications.
    The code for the data collection process utilized in this paper is publicly
  available at: https://github.com/microsoft/UFO/tree/main/dataflow, and
  comprehensive documentation can be found at
  https://microsoft.github.io/UFO/dataflow/overview/.
paper_ai_keywords: Large Language Models (LLMs) Large Action Models (LAMs) agent systems data collection model training environment integration grounding evaluation Windows OS-based agent