title: FLUX that Plays Music
summary: This paper explores a simple extension of diffusion-based rectified flow
  Transformers for text-to-music generation, termed as FluxMusic. Generally,
  along with design in advanced
  Fluxhttps://github.com/black-forest-labs/flux model, we transfers it
  into a latent VAE space of mel-spectrum. It involves first applying a sequence
  of independent attention to the double text-music stream, followed by a stacked
  single music stream for denoised patch prediction. We employ multiple
  pre-trained text encoders to sufficiently capture caption semantic information
  as well as inference flexibility. In between, coarse textual information, in
  conjunction with time step embeddings, is utilized in a modulation mechanism,
  while fine-grained textual details are concatenated with the music patch
  sequence as inputs. Through an in-depth study, we demonstrate that rectified
  flow training with an optimized architecture significantly outperforms
  established diffusion methods for the text-to-music task, as evidenced by
  various automatic metrics and human preference evaluations. Our experimental
  data, code, and model weights are made publicly available at:
  https://github.com/feizc/FluxMusic.
paper_authors._id: 66d7fd90a5098dc770352d7d
paper_authors.name: Zhengcong Fei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:38:48.396Z
paper_authors._id: 617ba1820e4237bd1731b867
paper_authors.avatarUrl: /avatars/f9de06363e64bddd7dc977e96e85df8a.svg
paper_authors.fullname: zhengcong fei
paper_authors.type: user
paper_authors.user: onion
paper_authors._id: 66d7fd90a5098dc770352d7e
paper_authors.name: Mingyuan Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:38:59.460Z
paper_authors._id: 634672bfb7b4e71c7f45360f
paper_authors.avatarUrl: /avatars/4b646fc3e271be90b9ec619d42ce3e99.svg
paper_authors.fullname: Fan Mingyuan
paper_authors.type: user
paper_authors.user: MichaelFan
paper_authors._id: 66d7fd90a5098dc770352d7f
paper_authors.name: Changqian Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:39:08.151Z
paper_authors._id: 6419e0b3ed725fef6444f53a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6419e0b3ed725fef6444f53a/3ZzIafBJa50-gAu1PASuG.png
paper_authors.fullname: Yu Changqian
paper_authors.type: user
paper_authors.user: Changqian
paper_authors._id: 66d7fd90a5098dc770352d80
paper_authors.name: Junshi Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:39:18.529Z
paper_authors._id: 642e8321be01b88c9447b25e
paper_authors.avatarUrl: /avatars/cc760d9b1bd943da1e6d21f14cc6d495.svg
paper_authors.fullname: HuangJunshi
paper_authors.type: user
paper_authors.user: huangjunshi
paper_title: FLUX that Plays Music
paper_summary: This paper explores a simple extension of diffusion-based rectified flow
  Transformers for text-to-music generation, termed as FluxMusic. Generally,
  along with design in advanced
  Fluxhttps://github.com/black-forest-labs/flux model, we transfers it
  into a latent VAE space of mel-spectrum. It involves first applying a sequence
  of independent attention to the double text-music stream, followed by a stacked
  single music stream for denoised patch prediction. We employ multiple
  pre-trained text encoders to sufficiently capture caption semantic information
  as well as inference flexibility. In between, coarse textual information, in
  conjunction with time step embeddings, is utilized in a modulation mechanism,
  while fine-grained textual details are concatenated with the music patch
  sequence as inputs. Through an in-depth study, we demonstrate that rectified
  flow training with an optimized architecture significantly outperforms
  established diffusion methods for the text-to-music task, as evidenced by
  various automatic metrics and human preference evaluations. Our experimental
  data, code, and model weights are made publicly available at:
  https://github.com/feizc/FluxMusic.
paper_ai_keywords: diffusion-based rectified flow Transformers text-to-music generation FluxMusic latent VAE mel-spectrum independent attention denoised patch prediction pre-trained text encoders caption semantic information modulation mechanism time step embeddings fine-grained textual details music patch sequence rectified flow training diffusion methods automatic metrics human preference evaluations