language, granularity of annotation, and text structure (e.g., some data is structured while others are
unstructured). To train a network for different tasks, simply mixing these diverse datasets cannot lead
to mutual enhancement; instead, it introduces interference. Most existing multi-task training approaches
have either grouped similar tasks (e.g., audio captioning, transcription) or assigned a dataset ID to each
dataset (Wang et al., 2023a; Lyu et al., 2023; Wu et al., 2023b; Gong et al., 2023b; Shu et al., 2023) to avoid
interference. Although these approaches have achieved certain effectiveness, there is still considerable
room for improvement. Whisper proposes a multitask training format by specifying tasks and condition
information as a sequence of input special tokens to the language decoder, such as voice activity detection,
language identification, and sentence-level timestamp tags. However, Whisper focuses on speech translation
and recognition tasks only.
Qwen-Audio