Augmenting text for spoken language understanding with Large Language
  Models
Spoken semantic parsing (SSP) involves generating machine-comprehensible
parses from input speech. Training robust models for existing application
domains represented in training data or extending to new domains requires
corresponding triplets of speech-transcript-semantic parse data, which is
expensive to obtain. In this paper, we address this challenge by examining
methods that can use transcript-semantic parse data (unpaired text) without
corresponding speech. First, when unpaired text is drawn from existing textual
corpora, Joint Audio Text (JAT) and Text-to-Speech (TTS) are compared as ways
to generate speech representations for unpaired text. Experiments on the STOP
dataset show that unpaired text from existing and new domains improves
performance by 2% and 30% in absolute Exact Match (EM) respectively. Second, we
consider the setting when unpaired text is not available in existing textual
corpora. We propose to prompt Large Language Models (LLMs) to generate unpaired
text for existing and new domains. Experiments show that examples and words
that co-occur with intents can be used to generate unpaired text with Llama
2.0. Using the generated text with JAT and TTS for spoken semantic parsing
improves EM on STOP by 1.4% and 2.6% absolute for existing and new domains
respectively.
65090eda948ce5dce89bb87f
Roshan Sharma
65090eda948ce5dce89bb880
Suyoun Kim
65090eda948ce5dce89bb881
Daniel Lazar
65090eda948ce5dce89bb882
Trang Le
65090eda948ce5dce89bb883
Akshat Shrivastava
65090eda948ce5dce89bb884
Kwanghoon Ahn
65090eda948ce5dce89bb885
Piyush Kansal
65090eda948ce5dce89bb886
Leda Sari
65090eda948ce5dce89bb887
Ozlem Kalinli
65090eda948ce5dce89bb888
Michael Seltzer
Augmenting text for spoken language understanding with Large Language
  Models
Spoken semantic parsing (SSP) involves generating machine-comprehensible
parses from input speech. Training robust models for existing application
domains represented in training data or extending to new domains requires
corresponding triplets of speech-transcript-semantic parse data, which is
expensive to obtain. In this paper, we address this challenge by examining
methods that can use transcript-semantic parse data (unpaired text) without
corresponding speech. First, when unpaired text is drawn from existing textual
corpora, Joint Audio Text (JAT) and Text-to-Speech (TTS) are compared as ways
to generate speech representations for unpaired text. Experiments on the STOP
dataset show that unpaired text from existing and new domains improves
performance by 2% and 30% in absolute Exact Match (EM) respectively. Second, we
consider the setting when unpaired text is not available in existing textual
corpora. We propose to prompt Large Language Models (LLMs) to generate unpaired
text for existing and new domains. Experiments show that examples and words
that co-occur with intents can be used to generate unpaired text with Llama
2.0. Using the generated text with JAT and TTS for spoken semantic parsing
improves EM on STOP by 1.4% and 2.6% absolute for existing and new domains
respectively.
Methods using transcript-semantic parse data without corresponding speech improve spoken semantic parsing performance, especially for new domains, through Joint Audio Text and Text-to-Speech techniques, and by prompting Large Language Models to generate unpaired text.
Joint Audio Text (JAT) Text-to-Speech (TTS) Large Language Models (LLMs) spoken semantic parsing transcript-semantic parse data Exact Match (EM) STOP dataset unpaired text