readme: ---
  language: vi
  datasets:
  - nguyenvulebinh/song_dataset
  tags:
  - speech
  license: cc-by-nc-4.0
  ---
  
  
  # Lyric alignment
  Vietnamese song lyric alignment framework
  
  ## Task description (Zalo AI challenge 2022)
  
  Many of us love to sing along with songs in the way of our favorite singers in albums (karaoke style). The target is building a model to align lyrics with a music audio.
  
  - Input: a music segment (including vocal) and its lyrics.
  
  - Output: start-time and end-time of each word in the lyrics.
  
  For evaluation, the accuracy of prediction will be evaluated using Intersection over Union ($IoU$).
  With $IoU$ metric, the higher the better. For example:
  
  ![](./assets/eval_example.png)
  
  $IoU$ of prediction and the ground truth of an audio segment $S_i$ is computed by the following formula: 
  
  > $IoU(S_i) = \frac{1}{m} \sum_{j=1}^{m}{\frac{G_j\cap P_j}{G_j\cup P_j}}$
  
  where $m$ is the number of tokens of $S_i$. Then the Final IoU of across all $n$ audio segments is the average of their corresponding $IoU$: 
  
  > $Final\_IoU = \frac{1}{n} \sum_{i=1}^{n}{IoU(S_i)}$
  
  ## Data description
  
  ### Zalo public dataset
  
  - Training data: 1057 music segments from ~ 480 songs. Each segment is provided with an audio formatted as WAV file and a ground-truth JSON file which includes lyrics and aligned time frame (in milliseconds) of each single word.
  
  - Testing data: 
      - Public test: 264 music segments from ~ 120 songs.
  
      - Private test: 464 music segments from ~ 200 songs.
  
  An example of data:
  
  <img src="./assets/sample_example.png" width="300" />
  
  ### Crawling public dataset
  
  Since the dataset provided by Zalo is small and noisy, we decided to crawl data from other public sources. Luckily, our strategies (detail in **Methodology** section) for this task do not need an aligned time frame for every single word but only the song and its lyric, just like a typical ASR dataset.
  
  We detail data crawling and processing in the [data_preparation](./data_preparation/README.md) folder. We crawled a total of 30.000 songs from https://zingmp3.vn website, that around 1.500 hours of audio.
  
  ## Methodology
  
  Our strategies are heavily based on the study of [CTC-Segmentation](https://arxiv.org/abs/2007.09127) by Ludwig Kürzinger and Pytorch tutorial of [Forced Alignment with Wav2Vec2](https://pytorch.org/audio/main/tutorials/forced_alignment_tutorial.html). Quote from Ludwig Kürzinger's study:
  
  > CTC-segmentation, an algorithm to extract proper audio-text alignments in the presence of additional unknown speech sections at the beginning or end of the audio recording. It uses a CTC-based end to-end network that was trained on already aligned data beforehand, e.g., as provided by a CTC/attention ASR system.
  
  Based on Pytorch tutorial of [Forced Alignment with Wav2Vec2](https://pytorch.org/audio/main/tutorials/forced_alignment_tutorial.html). The process of alignment looks like the following:
  
  1. Estimate the frame-wise label probability from audio waveform
  ![](https://pytorch.org/audio/main/_images/sphx_glr_forced_alignment_tutorial_001.png)
  2. Generate the trellis matrix which represents the probability of labels aligned at time step.
  ![](https://pytorch.org/audio/main/_images/sphx_glr_forced_alignment_tutorial_002.png)
  3. Find the most likely path from the trellis matrix.
  ![](https://pytorch.org/audio/main/_images/sphx_glr_forced_alignment_tutorial_005.png)
  
  The alignment only works well if either having good frame-wise probability and the correct label. 
  
  - A good frame-wise probability can be achieved from a robust acoustic model. Our setup's acoustic model is based on wav2vec2 architecture trained using CTC loss.
  
  - A correct label mean the spoken form label. Because of lyric came from diverse of source, it can  include special characters, mix English and Vietnamese word, number format (date, time, currency,...), nickname, ... This kind of data will make the model hard to map between audio signal and text lyric. Our soulution is mapping all word of lyric from written form to spoken form. For example:
  
      | Written                                           | Spoken      |
      |--------------------------------------------------|--------------|
      | joker | giốc cơ |
      | running                  | răn ninh    |
      | 0h   | không giờ   |
  
      To convert English words to pronunciation way in Vietnamese, we use [nguyenvulebinh/spelling-oov](
      https://huggingface.co/nguyenvulebinh/spelling-oov) model. For handling number format, we use [Vinorm](https://github.com/v-nhandt21/Vinorm). For other special characters ".,?...", we delete it.
  
      The final time alignment of a written word (e.g. 0h) is a concatenated time alignment of its spoken words (e.g. không giờ).
  
  ## Evaluation setup
  
  ### Acoustic model
  
  Our final model is based on [nguyenvulebinh/wav2vec2-large-vi-vlsp2020](https://huggingface.co/nguyenvulebinh/wav2vec2-large-vi-vlsp2020) model. It pre-trained on 13k hours of Vietnamese youtube audio (un-label data) and fine-tuned on 250 hour