title: LLaSM: Large Language and Speech Model
summary: Multi-modal large language models have garnered significant interest
  recently. Though, most of the works focus on vision-language multi-modal models
  providing strong capabilities in following vision-and-language instructions.
  However, we claim that speech is also an important modality through which
  humans interact with the world. Hence, it is crucial for a general-purpose
  assistant to be able to follow multi-modal speech-and-language instructions. In
  this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an
  end-to-end trained large multi-modal speech-language model with cross-modal
  conversational abilities, capable of following speech-and-language
  instructions. Our early experiments show that LLaSM demonstrates a more
  convenient and natural way for humans to interact with artificial intelligence.
  Specifically, we also release a large Speech Instruction Following dataset
  LLaSM-Audio-Instructions. Code and demo are available at
  https://github.com/LinkSoul-AI/LLaSM and
  https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions
  dataset is available at
  https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.
paper_authors._id: 64f00175c26b0228ec52ab3b
paper_authors.name: Yu Shu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-31T12:05:38.870Z
paper_authors._id: 63842c734c48969dcf1d350b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63842c734c48969dcf1d350b/zl41G0gGjyIqY7w1ARdrB.jpeg
paper_authors.fullname: shu yu
paper_authors.type: user
paper_authors.user: TabShu
paper_authors._id: 64f00175c26b0228ec52ab3c
paper_authors.name: Siwei Dong
paper_authors._id: 64f00175c26b0228ec52ab3d
paper_authors.name: Guangyao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-31T07:15:40.133Z
paper_authors._id: 6462de3277f63463fc60ef09
paper_authors.avatarUrl: /avatars/d6418be979c06a37df6c5c8e3c6ae235.svg
paper_authors.fullname: Guangyao Chen
paper_authors.type: user
paper_authors.user: iCGY96
paper_authors._id: 64f00175c26b0228ec52ab3e
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:32:06.858Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 64f00175c26b0228ec52ab3f
paper_authors.name: Ruihua Zhang
paper_authors._id: 64f00175c26b0228ec52ab40
paper_authors.name: Daochen Shi
paper_authors._id: 64f00175c26b0228ec52ab41
paper_authors.name: Qiqi Xiang
paper_authors._id: 64f00175c26b0228ec52ab42
paper_authors.name: Yemin Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-31T12:03:38.277Z
paper_authors._id: 6310833434c7d77420b0a000
paper_authors.avatarUrl: /avatars/57fc8c667691ff6842c23a9389b16169.svg
paper_authors.fullname: Yemin
paper_authors.type: user
paper_authors.user: shiyemin2
paper_title: LLaSM: Large Language and Speech Model
paper_summary: Multi-modal large language models have garnered significant interest
  recently. Though, most of the works focus on vision-language multi-modal models
  providing strong capabilities in following vision-and-language instructions.
  However, we claim that speech is also an important modality through which
  humans interact with the world. Hence, it is crucial for a general-purpose
  assistant to be able to follow multi-modal speech-and-language instructions. In
  this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an
  end-to-end trained large multi-modal speech-language model with cross-modal
  conversational abilities, capable of following speech-and-language
  instructions. Our early experiments show that LLaSM demonstrates a more
  convenient and natural way for humans to interact with artificial intelligence.
  Specifically, we also release a large Speech Instruction Following dataset
  LLaSM-Audio-Instructions. Code and demo are available at
  https://github.com/LinkSoul-AI/LLaSM and
  https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions
  dataset is available at
  https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.
paper_ai_summary: LLaSM, an end-to-end trained large multi-modal speech-language model, enhances human interaction with AI by following speech-and-language instructions using cross-modal conversational abilities.
paper_ai_keywords: multi-modal large language models vision-language multi-modal models LLaSM cross-modal conversational abilities Speech Instruction Following dataset LLaSM-Audio-Instructions