Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning
<|reference_start|>Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning: As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations. In this work, we propose Voicebox Adapter, a novel approach that integrates fine-grained conditions into a pre-trained Voicebox speech generation model using a cross-attention module. To ensure a smooth integration of newly added modules with pre-trained ones, we explore various efficient fine-tuning approaches. Our experiment shows that the LoRA with bias-tuning configuration yields the best performance, enhancing controllability without compromising speech quality. Across three fine-grained conditional generation tasks, we demonstrate the effectiveness and resource efficiency of Voicebox Adapter. Follow-up experiments further highlight the robustness of Voicebox Adapter across diverse data setups.<|reference_end|>
@article{chien2024learning,
    title={Learning Fine-Grained Controllability on Speech Generation via Efficient
  Fine-Tuning},
    author={Chung-Ming Chien, Andros Tjandra, Apoorv Vyas, Matt Le, Bowen Shi,
  Wei-Ning Hsu},
    journal={arXiv preprint arXiv:2406.06251},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.06251},
    primaryClass={eess.AS cs.CL}
}