title: SpiRit-LM: Interleaved Spoken and Written Language Model
summary: We introduce SPIRIT-LM, a foundation multimodal language model that freely
  mixes text and speech. Our model is based on a pretrained text language model
  that we extend to the speech modality by continuously training it on text and
  speech units. Speech and text sequences are concatenated as a single set of
  tokens, and trained with a word-level interleaving method using a small
  automatically-curated speech-text parallel corpus. SPIRIT-LM comes in two
  versions: a BASE version that uses speech semantic units and an EXPRESSIVE
  version that models expressivity using pitch and style units in addition to the
  semantic units. For both versions, the text is encoded with subword BPE tokens.
  The resulting model displays both the semantic abilities of text models and the
  expressive abilities of speech models. Additionally, we demonstrate that
  SPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities
  (i.e. ASR, TTS, Speech Classification).
paper_authors._id: 65c5984702262fe3b4fb9c57
paper_authors.name: Tu Anh Nguyen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:17:00.027Z
paper_authors._id: 62e7b942c9e83245ea5e14ec
paper_authors.avatarUrl: /avatars/3223dae7ca6b80e6dc780c1b002b1524.svg
paper_authors.fullname: Tu Anh Nguyen
paper_authors.type: user
paper_authors.user: ntuanh
paper_authors._id: 65c5984702262fe3b4fb9c58
paper_authors.name: Benjamin Muller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:17:06.496Z
paper_authors._id: 5e6b7153d4cd9779932a7600
paper_authors.avatarUrl: /avatars/c1304f100758c5e2a54fd4d47f638841.svg
paper_authors.fullname: Benjamin Muller
paper_authors.type: user
paper_authors.user: benjamin-mlr
paper_authors._id: 65c5984702262fe3b4fb9c59
paper_authors.name: Bokai Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:17:26.496Z
paper_authors._id: 65b3c0ee05c25412bb6aab2c
paper_authors.avatarUrl: /avatars/2b3be41c899a29e2a0954dd7cd6cd5ab.svg
paper_authors.fullname: Bokai Yu
paper_authors.type: user
paper_authors.user: bokaiyu
paper_authors._id: 65c5984702262fe3b4fb9c5a
paper_authors.name: Marta R. Costa-jussa
paper_authors._id: 65c5984702262fe3b4fb9c5b
paper_authors.name: Maha Elbayad
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:17:42.732Z
paper_authors._id: 64e243dff8934329dd72098f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e243dff8934329dd72098f/JbB4zvlendHSRXjx_iC0j.jpeg
paper_authors.fullname: Maha Elbayad
paper_authors.type: user
paper_authors.user: elbayadm
paper_authors._id: 65c5984702262fe3b4fb9c5c
paper_authors.name: Sravya Popuri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:17:48.409Z
paper_authors._id: 62e190f3926f4892a4c8099e
paper_authors.avatarUrl: /avatars/2168f769c526362f9ed35473f4b30047.svg
paper_authors.fullname: Sravya Popuri
paper_authors.type: user
paper_authors.user: sravyapopuri388
paper_authors._id: 65c5984702262fe3b4fb9c5d
paper_authors.name: Paul-Ambroise Duquenne
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:17:54.553Z
paper_authors._id: 6331a09fe092098b57bc1d83
paper_authors.avatarUrl: /avatars/cdbc4be17c160b5e1f957b6c1ae29e22.svg
paper_authors.fullname: Paul-Ambroise Duquenne
paper_authors.type: user
paper_authors.user: padqn
paper_authors._id: 65c5984702262fe3b4fb9c5e
paper_authors.name: Robin Algayres
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:18:00.654Z
paper_authors._id: 64d25f2035577e62a9df31a8
paper_authors.avatarUrl: /avatars/fdaeb527f084d21e35de5a625c3629d6.svg
paper_authors.fullname: robin algayres
paper_authors.type: user
paper_authors.user: robinalgayres
paper_authors._id: 65c5984702262fe3b4fb9c5f
paper_authors.name: Ruslan Mavlyutov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:18:06.330Z
paper_authors._id: 64dcfecb249785efad9a514d
paper_authors.avatarUrl: /avatars/77de68494021f298bf2561b2ea9ffef8.svg
paper_authors.fullname: Ruslan Mavlyutov
paper_authors.type: user
paper_authors.user: mavlyutovrus
paper_authors._id: 65c5984702262fe3b4fb9c60
paper_authors.name: Itai Gat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:18:12.797Z
paper_authors._id: 62a8fa984d933c74bf410c16
paper_authors.avatarUrl: /avatars/73519deba3176be9c23d49f749aee5da.svg
paper_authors.fullname: Itai Gat
paper_authors.type: user
paper_authors.user: itaigat
paper_authors._id: 65c5984702262fe3b4fb9c61
paper_authors.name: Gabriel Synnaeve
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:18:19.465Z
paper_authors._id: 630eac7931970d1cd4fbacf2
paper_authors.avatarUrl: /avatars/b7ccbddfa745db854dc342be1327cd53.svg
paper_authors.fullname: Gabriel Synnaeve
paper_authors.type: user
paper_authors.user: gsynnaeve
paper_authors._id: 65c5984702262fe3b4fb9c62
paper_authors.name: Juan Pino
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:18:25.364Z
paper_authors._id: 634deff6be5a827d4874546a
paper_authors.avatarUrl: /avatars/05e0ad93d9c8c7e572c6f73feb649aeb.svg
paper_authors.fullname: Juan Pino
paper_authors.type: user
paper_authors.user: juanpino
paper_authors._id: 65c5984702262fe3b4fb9c63
paper_authors.name: Benoit Sagot
paper_authors._id: 65c5984702262fe3b4fb9c64
paper_authors.name: Emmanuel Dupoux
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:18:34.382Z
paper_authors._id: 63317d2118711776b4663c3a
paper_authors.avatarUrl: /avatars/7dedd1934c1000b6f81a2a37ec348347.svg
paper_authors.fullname: Emmanuel Dupoux
paper_authors.type: user
paper_authors.user: edupoux
paper_title: SpiRit-LM: Interleaved Spoken and Written Language Model
paper_summary: We introduce SPIRIT-LM, a foundation multimodal language model that freely
  mixes text and speech. Our model is based on a pretrained text language model
  that we extend to the speech modality by continuously training it on text and
  speech units. Speech and text sequences are concatenated as a single set of
  tokens, and trained with a word-level interleaving method using a small
  automatically-curated speech-text parallel corpus. SPIRIT-LM comes in two
  versions: a BASE version that uses speech semantic units and an EXPRESSIVE
  version that models expressivity using pitch and style units in addition to the
  semantic units. For both versions, the text is encoded with subword BPE tokens.
  The resulting model displays both the semantic abilities of text models and the
  expressive abilities of speech models. Additionally, we demonstrate that
  SPIRIT-LM is able to learn new tasks in a few-shot fashion across modalities
  (i.e. ASR, TTS, Speech Classification).
paper_ai_keywords: multimodal language model pretrained text language model speech modality speech units word-level interleaving subword BPE tokens speech semantic units pitch and style units few-shot learning ASR (Automatic Speech Recognition) TTS (Text-to-Speech) Speech Classification