title: Zipper: A Multi-Tower Decoder Architecture for Fusing Modalities
summary: Integrating multiple generative foundation models, especially those trained
  on different modalities, into something greater than the sum of its parts poses
  significant challenges. Two key hurdles are the availability of aligned data
  (concepts that contain similar meaning but is expressed differently in
  different modalities), and effectively leveraging unimodal representations in
  cross-domain generative tasks, without compromising their original unimodal
  capabilities.
    We propose Zipper, a multi-tower decoder architecture that addresses these
  concerns by using cross-attention to flexibly compose multimodal generative
  models from independently pre-trained unimodal decoders. In our experiments
  fusing speech and text modalities, we show the proposed architecture performs
  very competitively in scenarios with limited aligned text-speech data. We also
  showcase the flexibility of our model to selectively maintain unimodal (e.g.,
  text-to-text generation) generation performance by freezing the corresponding
  modal tower (e.g. text). In cross-modal tasks such as automatic speech
  recognition (ASR) where the output modality is text, we show that freezing the
  text backbone results in negligible performance degradation. In cross-modal
  tasks such as text-to-speech generation (TTS) where the output modality is
  speech, we show that using a pre-trained speech backbone results in superior
  performance to the baseline.
paper_authors._id: 665888130a93bd5f0afe9a83
paper_authors.name: Vicky Zayats
paper_authors._id: 665888130a93bd5f0afe9a84
paper_authors.name: Peter Chen
paper_authors._id: 665888130a93bd5f0afe9a85
paper_authors.name: Melissa Merrari
paper_authors._id: 665888130a93bd5f0afe9a86
paper_authors.name: Dirk Padfield
paper_title: Zipper: A Multi-Tower Decoder Architecture for Fusing Modalities
paper_summary: Integrating multiple generative foundation models, especially those trained
  on different modalities, into something greater than the sum of its parts poses
  significant challenges. Two key hurdles are the availability of aligned data
  (concepts that contain similar meaning but is expressed differently in
  different modalities), and effectively leveraging unimodal representations in
  cross-domain generative tasks, without compromising their original unimodal
  capabilities.
    We propose Zipper, a multi-tower decoder architecture that addresses these
  concerns by using cross-attention to flexibly compose multimodal generative
  models from independently pre-trained unimodal decoders. In our experiments
  fusing speech and text modalities, we show the proposed architecture performs
  very competitively in scenarios with limited aligned text-speech data. We also
  showcase the flexibility of our model to selectively maintain unimodal (e.g.,
  text-to-text generation) generation performance by freezing the corresponding
  modal tower (e.g. text). In cross-modal tasks such as automatic speech
  recognition (ASR) where the output modality is text, we show that freezing the
  text backbone results in negligible performance degradation. In cross-modal
  tasks such as text-to-speech generation (TTS) where the output modality is
  speech, we show that using a pre-trained speech backbone results in superior
  performance to the baseline.
paper_ai_summary: Zipper, a multi-tower decoder using cross-attention, integrates unimodal generative models for effective cross-modal tasks with limited aligned data while maintaining unimodal performance.
paper_ai_keywords: generative foundation models cross-attention multi-tower decoder aligned data unimodal representations cross-domain generative tasks text-to-text generation automatic speech recognition text-to-speech generation pre-trained speech backbone