Large Action Models: From Inception to Implementation
As AI continues to advance, there is a growing demand for systems that go
beyond language-based assistance and move toward intelligent agents capable of
performing real-world actions. This evolution requires the transition from
traditional Large Language Models (LLMs), which excel at generating textual
responses, to Large Action Models (LAMs), designed for action generation and
execution within dynamic environments. Enabled by agent systems, LAMs hold the
potential to transform AI from passive language understanding to active task
completion, marking a significant milestone in the progression toward
artificial general intelligence.
  In this paper, we present a comprehensive framework for developing LAMs,
offering a systematic approach to their creation, from inception to deployment.
We begin with an overview of LAMs, highlighting their unique characteristics
and delineating their differences from LLMs. Using a Windows OS-based agent as
a case study, we provide a detailed, step-by-step guide on the key stages of
LAM development, including data collection, model training, environment
integration, grounding, and evaluation. This generalizable workflow can serve
as a blueprint for creating functional LAMs in various application domains. We
conclude by identifying the current limitations of LAMs and discussing
directions for future research and industrial deployment, emphasizing the
challenges and opportunities that lie ahead in realizing the full potential of
LAMs in real-world applications.
  The code for the data collection process utilized in this paper is publicly
available at: https://github.com/microsoft/UFO/tree/main/dataflow, and
comprehensive documentation can be found at
https://microsoft.github.io/UFO/dataflow/overview/.
675f9dce5dca7e3c5c1f0950
Lu Wang
675f9dce5dca7e3c5c1f0951
Fangkai Yang
claimed_verified
2025-02-27T09:18:20.886Z
669dcf6200970c3b27aafa5d
/avatars/bb9ed5ff86326fdaeb184c6b0e40f74f.svg
kaikai yang
user
keanudicap
675f9dce5dca7e3c5c1f0952
Chaoyun Zhang
admin_assigned
2024-12-16T10:13:55.597Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
675f9dce5dca7e3c5c1f0953
Junting Lu
admin_assigned
2024-12-16T10:14:25.301Z
6724f7b2f0e98abd46cb94d3
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/O47r1efbW0MYxGyhixw9k.png
lu junior ting
user
Lujunting
675f9dce5dca7e3c5c1f0954
Jiaxu Qian
675f9dce5dca7e3c5c1f0955
Shilin He
admin_assigned
2024-12-16T10:14:11.939Z
62c6df026a092eda1f1ab6e5
/avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
Shilin He
user
shilhe
675f9dce5dca7e3c5c1f0956
Pu Zhao
675f9dce5dca7e3c5c1f0957
Bo Qiao
675f9dce5dca7e3c5c1f0958
Ray Huang
675f9dce5dca7e3c5c1f0959
Si Qin
675f9dce5dca7e3c5c1f095a
Qisheng Su
675f9dce5dca7e3c5c1f095b
Jiayi Ye
675f9dce5dca7e3c5c1f095c
Yudi Zhang
claimed_verified
2024-12-17T08:04:14.067Z
662a5511be52f6c4d5dd0b64
/avatars/759686e916c36ebea5908e400d1cf25b.svg
Yudi Zhang
user
ReedZyd
675f9dce5dca7e3c5c1f095d
Jian-Guang Lou
admin_assigned
2024-12-16T10:11:51.046Z
6554338ba4de813482e91ad5
/avatars/bee879fa4a818fd6c070ee001769ce05.svg
Jian-Guang Lou
user
substill
675f9dce5dca7e3c5c1f095e
Qingwei Lin
admin_assigned
2024-12-16T10:11:45.046Z
652fc9f39bc50a6c0e435224
https://cdn-avatars.huggingface.co/v1/production/uploads/652fc9f39bc50a6c0e435224/70OBVDHHBsxG2giJ-E3_1.jpeg
Lin Qingwei
user
Eliblo1969
675f9dce5dca7e3c5c1f095f
Saravan Rajmohan
675f9dce5dca7e3c5c1f0960
Dongmei Zhang
admin_assigned
2024-12-16T10:11:13.362Z
66473d2c7abe6ad66e81a3dd
/avatars/82f40244806c06ffeaa1c4265e9725ea.svg
ZHANGDONGMEI
user
ZDM6426
675f9dce5dca7e3c5c1f0961
Qi Zhang
Large Action Models: From Inception to Implementation
As AI continues to advance, there is a growing demand for systems that go
beyond language-based assistance and move toward intelligent agents capable of
performing real-world actions. This evolution requires the transition from
traditional Large Language Models (LLMs), which excel at generating textual
responses, to Large Action Models (LAMs), designed for action generation and
execution within dynamic environments. Enabled by agent systems, LAMs hold the
potential to transform AI from passive language understanding to active task
completion, marking a significant milestone in the progression toward
artificial general intelligence.
  In this paper, we present a comprehensive framework for developing LAMs,
offering a systematic approach to their creation, from inception to deployment.
We begin with an overview of LAMs, highlighting their unique characteristics
and delineating their differences from LLMs. Using a Windows OS-based agent as
a case study, we provide a detailed, step-by-step guide on the key stages of
LAM development, including data collection, model training, environment
integration, grounding, and evaluation. This generalizable workflow can serve
as a blueprint for creating functional LAMs in various application domains. We
conclude by identifying the current limitations of LAMs and discussing
directions for future research and industrial deployment, emphasizing the
challenges and opportunities that lie ahead in realizing the full potential of
LAMs in real-world applications.
  The code for the data collection process utilized in this paper is publicly
available at: https://github.com/microsoft/UFO/tree/main/dataflow, and
comprehensive documentation can be found at
https://microsoft.github.io/UFO/dataflow/overview/.
A framework is proposed for developing Large Action Models (LAMs) that generate and execute actions in dynamic environments, moving beyond traditional Large Language Models (LLMs) towards intelligent agents capable of task completion.
Large Action Models LAMs Large Language Models LLMs agent systems data collection model training environment integration grounding evaluation