title: WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words
summary: Training on multiple modalities of input can augment the capabilities of a
  language model. Here, we ask whether such a training regime can improve the
  quality and efficiency of these systems as well. We focus on text--audio and
  introduce Whisbert, which is inspired by the text--image approach of FLAVA
  singh_flava_2022. In accordance with Babylm warstadt2023papers
  guidelines, we pretrain Whisbert on a dataset comprising only 100 million words
  plus their corresponding speech from the word-aligned version of the People's
  Speech dataset galvez_peoples_2021. To assess the impact of
  multimodality, we compare versions of the model that are trained on text only
  and on both audio and text simultaneously. We find that while Whisbert is able
  to perform well on multimodal masked modeling and surpasses the Babylm
  baselines in most benchmark tasks, it struggles to optimize its complex
  objective and outperform its text-only Whisbert baseline.
paper_authors._id: 656fd6d77f50602340bea793
paper_authors.name: Lukas Wolf
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-07T08:30:04.253Z
paper_authors._id: 63ea0c6e2e476376eb08b9fe
paper_authors.avatarUrl: /avatars/a92b5261bd090f142945b60bdba19714.svg
paper_authors.fullname: Lukas 
paper_authors.type: user
paper_authors.user: luwo
paper_authors._id: 656fd6d77f50602340bea794
paper_authors.name: Klemen Kotar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:11:05.340Z
paper_authors._id: 646d3cae4220471ca0c6cb13
paper_authors.avatarUrl: /avatars/30e47db03a23d49d946cc5248b28959a.svg
paper_authors.fullname: Klemen Kotar
paper_authors.type: user
paper_authors.user: klemenk
paper_authors._id: 656fd6d77f50602340bea795
paper_authors.name: Greta Tuckute
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-12-06T02:18:43.200Z
paper_authors._id: 605a5f796ce6cabbb3474b55
paper_authors.avatarUrl: /avatars/971866aa240126e480bbe01eff64c009.svg
paper_authors.fullname: Greta Tuckute
paper_authors.type: user
paper_authors.user: gretatuckute
paper_authors._id: 656fd6d77f50602340bea796
paper_authors.name: Eghbal Hosseini
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-12-06T02:05:11.794Z
paper_authors._id: 61de0406c59d602a3a85b186
paper_authors.avatarUrl: /avatars/b95a77d962f7219bea7a397b272af792.svg
paper_authors.fullname: Eghbal Hosseini
paper_authors.type: user
paper_authors.user: ehoseini
paper_authors._id: 656fd6d77f50602340bea797
paper_authors.name: Tamar Regev
paper_authors._id: 656fd6d77f50602340bea798
paper_authors.name: Ethan Wilcox
paper_authors._id: 656fd6d77f50602340bea799
paper_authors.name: Alex Warstadt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:11:41.067Z
paper_authors._id: 5ee772b8464d0272c8b24560
paper_authors.avatarUrl: /avatars/81dfd862d13696936ed05e8ad966f926.svg
paper_authors.fullname: Alex Warstadt
paper_authors.type: user
paper_authors.user: alexwarstadt
paper_title: WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words
paper_summary: Training on multiple modalities of input can augment the capabilities of a
  language model. Here, we ask whether such a training regime can improve the
  quality and efficiency of these systems as well. We focus on text--audio and
  introduce Whisbert, which is inspired by the text--image approach of FLAVA
  singh_flava_2022. In accordance with Babylm warstadt2023papers
  guidelines, we pretrain Whisbert on a dataset comprising only 100 million words
  plus their corresponding speech from the word-aligned version of the People's
  Speech dataset galvez_peoples_2021. To assess the impact of
  multimodality, we compare versions of the model that are trained on text only
  and on both audio and text simultaneously. We find that while Whisbert is able
  to perform well on multimodal masked modeling and surpasses the Babylm
  baselines in most benchmark tasks, it struggles to optimize its complex
  objective and outperform its text-only Whisbert baseline.
paper_ai_summary: A multimodal text--audio language model, Whisbert, based on the FLAVA approach, shows improved performance on multimodal tasks but does not consistently outperform its text-only counterpart.
paper_ai_keywords: multimodality text--audio Whisbert FLAVA Babylm pretraining People's Speech dataset multimodal masked modeling