Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach
<|reference_start|>Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach: Recent progress in Spoken Language Modeling has demonstrated the feasibility of learning language directly from speech. Generating speech through a pipeline that operates at the text level typically loses nuances, intonations, and non-verbal vocalizations. Modeling directly from speech opens up the path to more natural and expressive systems. On the other hand, speech-only systems tend to trail behind text-based language models in terms of their semantic abilities. We show that fine-tuning speech representation models on phoneme classification leads to more context-invariant representations, which in turn improve downstream language modeling performance.<|reference_end|>
@article{poli2024improving,
    title={Improving Spoken Language Modeling with Phoneme Classification: A Simple
  Fine-tuning Approach},
    author={Maxime Poli, Emmanuel Chemla, Emmanuel Dupoux},
    journal={arXiv preprint arXiv:2410.00025},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00025},
    primaryClass={cs.CL cs.SD eess.AS}
}