AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
We introduce AnyGPT, an any-to-any multimodal language model that utilizes
discrete representations for the unified processing of various modalities,
including speech, text, images, and music. AnyGPT can be trained stably without
any alterations to the current large language model (LLM) architecture or
training paradigms. Instead, it relies exclusively on data-level preprocessing,
facilitating the seamless integration of new modalities into LLMs, akin to the
incorporation of new languages. We build a multimodal text-centric dataset for
multimodal alignment pre-training. Utilizing generative models, we synthesize
the first large-scale any-to-any multimodal instruction dataset. It consists of
108k samples of multi-turn conversations that intricately interweave various
modalities, thus equipping the model to handle arbitrary combinations of
multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is
capable of facilitating any-to-any multimodal conversation while achieving
performance comparable to specialized models across all modalities, proving
that discrete representations can effectively and conveniently unify multiple
modalities within a language model. Demos are shown in
https://junzhan2000.github.io/AnyGPT.github.io/
65d441b42f7b897bfaa26089
Jun Zhan
admin_assigned
2024-02-20T09:34:18.222Z
6509858fa2abcb18d633597b
https://cdn-avatars.huggingface.co/v1/production/uploads/6509858fa2abcb18d633597b/uxfPgCvhL1Tzw_lDR9hW-.jpeg
JunZhan
user
zhanjun
65d441b42f7b897bfaa2608a
Junqi Dai
65d441b42f7b897bfaa2608b
Jiasheng Ye
65d441b42f7b897bfaa2608c
Yunhua Zhou
65d441b42f7b897bfaa2608d
Dong Zhang
admin_assigned
2024-02-20T09:36:12.891Z
6377596f82ce7c3b8a528f72
/avatars/f55676883e054d7e4a3b96ba1e752a80.svg
Dong Zhang
user
nutation
65d441b42f7b897bfaa2608e
Zhigeng Liu
65d441b42f7b897bfaa2608f
Xin Zhang
claimed_verified
2024-02-21T08:30:11.642Z
64eda039067fbb625f851207
/avatars/fb6014110c32ab0d46a63558d448e00d.svg
zhangxin
user
xin98
65d441b42f7b897bfaa26090
Ruibin Yuan
admin_assigned
2024-02-20T09:36:54.038Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
65d441b42f7b897bfaa26091
Ge Zhang
admin_assigned
2024-02-20T09:37:43.902Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
65d441b42f7b897bfaa26092
Linyang Li
claimed_verified
2024-07-18T09:09:27.606Z
6698ad87a9778d20ff373142
/avatars/5e3546d124e8c4f4636a41eab6f0cf68.svg
linyangli
user
linyangli
65d441b42f7b897bfaa26093
Hang Yan
65d441b42f7b897bfaa26094
Jie Fu
claimed_verified
2024-02-21T08:30:13.762Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
65d441b42f7b897bfaa26095
Tao Gui
65d441b42f7b897bfaa26096
Tianxiang Sun
admin_assigned
2024-02-20T09:39:28.011Z
618c9e4cf35289d4bcab9d02
/avatars/28b5b096190b539295ec0da167f1abf3.svg
Tianxiang Sun
user
txsun
65d441b42f7b897bfaa26097
Yugang Jiang
65d441b42f7b897bfaa26098
Xipeng Qiu
admin_assigned
2024-02-20T09:39:48.694Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
We introduce AnyGPT, an any-to-any multimodal language model that utilizes
discrete representations for the unified processing of various modalities,
including speech, text, images, and music. AnyGPT can be trained stably without
any alterations to the current large language model (LLM) architecture or
training paradigms. Instead, it relies exclusively on data-level preprocessing,
facilitating the seamless integration of new modalities into LLMs, akin to the
incorporation of new languages. We build a multimodal text-centric dataset for
multimodal alignment pre-training. Utilizing generative models, we synthesize
the first large-scale any-to-any multimodal instruction dataset. It consists of
108k samples of multi-turn conversations that intricately interweave various
modalities, thus equipping the model to handle arbitrary combinations of
multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is
capable of facilitating any-to-any multimodal conversation while achieving
performance comparable to specialized models across all modalities, proving
that discrete representations can effectively and conveniently unify multiple
modalities within a language model. Demos are shown in
https://junzhan2000.github.io/AnyGPT.github.io/
AnyGPT is a multimodal language model using discrete representations to process and generate content across various modalities with performance on par with specialized models.
discrete representations multimodal language model multimodal alignment pre-training generative models multimodal instruction dataset multi-turn conversations