LLaSM: Large Language and Speech Model
Multi-modal large language models have garnered significant interest
recently. Though, most of the works focus on vision-language multi-modal models
providing strong capabilities in following vision-and-language instructions.
However, we claim that speech is also an important modality through which
humans interact with the world. Hence, it is crucial for a general-purpose
assistant to be able to follow multi-modal speech-and-language instructions. In
this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an
end-to-end trained large multi-modal speech-language model with cross-modal
conversational abilities, capable of following speech-and-language
instructions. Our early experiments show that LLaSM demonstrates a more
convenient and natural way for humans to interact with artificial intelligence.
Specifically, we also release a large Speech Instruction Following dataset
LLaSM-Audio-Instructions. Code and demo are available at
https://github.com/LinkSoul-AI/LLaSM and
https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions
dataset is available at
https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.
64f00175c26b0228ec52ab3b
Yu Shu
admin_assigned
2023-08-31T12:05:38.870Z
63842c734c48969dcf1d350b
https://cdn-avatars.huggingface.co/v1/production/uploads/63842c734c48969dcf1d350b/zl41G0gGjyIqY7w1ARdrB.jpeg
shu yu
user
TabShu
64f00175c26b0228ec52ab3c
Siwei Dong
64f00175c26b0228ec52ab3d
Guangyao Chen
admin_assigned
2023-08-31T07:15:40.133Z
6462de3277f63463fc60ef09
/avatars/d6418be979c06a37df6c5c8e3c6ae235.svg
Guangyao Chen
user
iCGY96
64f00175c26b0228ec52ab3e
Wenhao Huang
claimed_verified
2024-10-13T20:32:06.858Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
64f00175c26b0228ec52ab3f
Ruihua Zhang
64f00175c26b0228ec52ab40
Daochen Shi
64f00175c26b0228ec52ab41
Qiqi Xiang
64f00175c26b0228ec52ab42
Yemin Shi
admin_assigned
2023-08-31T12:03:38.277Z
6310833434c7d77420b0a000
/avatars/57fc8c667691ff6842c23a9389b16169.svg
Yemin
user
shiyemin2
LLaSM: Large Language and Speech Model
Multi-modal large language models have garnered significant interest
recently. Though, most of the works focus on vision-language multi-modal models
providing strong capabilities in following vision-and-language instructions.
However, we claim that speech is also an important modality through which
humans interact with the world. Hence, it is crucial for a general-purpose
assistant to be able to follow multi-modal speech-and-language instructions. In
this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an
end-to-end trained large multi-modal speech-language model with cross-modal
conversational abilities, capable of following speech-and-language
instructions. Our early experiments show that LLaSM demonstrates a more
convenient and natural way for humans to interact with artificial intelligence.
Specifically, we also release a large Speech Instruction Following dataset
LLaSM-Audio-Instructions. Code and demo are available at
https://github.com/LinkSoul-AI/LLaSM and
https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions
dataset is available at
https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.
LLaSM, an end-to-end trained large multi-modal speech-language model, enhances human interaction with AI by following speech-and-language instructions using cross-modal conversational abilities.
multi-modal large language models vision-language multi-modal models LLaSM cross-modal conversational abilities Speech Instruction Following dataset LLaSM-Audio-Instructions