readme: # SUPERB Submission Template
  
  Welcome to the [SUPERB Challenge](https://superbbenchmark.org/challenge-slt2022/challenge_overview)! SUPERB is a collection of benchmarking resources to evaluate the capability of a universal shared representation for speech processing. It comes with a benchmark on the publicly available datasets and a challenge on a secret/not released hidden dataset. In SUPERB Challenge, a challenging hidden dataset is newly recorded to evaluate the ultimate generaliziblity across various tasks and data.
  
  You can participate the challenge by simply submitting your self-supervised (SSL) pretrained models (model definition & pretrained weights), and we benchmark it with the hidden datasets. This repository constains useful tools to let you easliy [submit](https://superbbenchmark.org/submit) your models ***privately*** for evaluation to [the challenge hidden-set leaderboard](https://superbbenchmark.org/leaderboard?track=constrained&subset=Hidden+Dev+Set).
  
  1. Generate a submission template
  2. Validate the format/interface correctness of your model
  3. Upload to Huggingface's Hub (privately)
  4. Submit the upload information to [SUPERB website](https://superbbenchmark.org/submit)
  
  #### Note 1.
  
  We accept pre-trained models in PyTorch by default. If you wish to submit upstreams in non-PyTorch frameworks, please mail to [superb.announcement@gmail.com](mailto:superb.announcement@gmail.com)!
  
  #### Note 2.
  
  If you are not feasible to submit the pre-trained model, please mail to [superb.announcement@gmail.com](mailto:superb.announcement@gmail.com) for us to see how to help!
  
  ## Quickstart
  
  ### 1. Add model interfaces
  
  #### forward
  
  Extract features from waveforms.
  
  - **Input:** A list of waveforms in 16000 Hz
  
      ```python
      SAMPLE_RATE = 16000
      BATCH_SIZE = 8
      EXAMPLE_SEC = 10
      wavs = [torch.randn(SAMPLE_RATE * EXAMPLE_SEC).cuda() for _ in range(BATCH_SIZE)]
      ```
  
  - **Output:** A dictionary with a key "hidden_states" (for compatiblility with old ver.). The value is **a list** of padded sequences in the same shape of **(batch_size, max_sequence_length_of_batch, hidden_size)** for weighted-sum to work. It is welcome to perform some task-specified / independent pre- / post-processing on the upstream's raw hidden-sets, including upsampling and downsampling. However, all the values must come from **a single upstream model**:
  
      ```python
      tasks = ["hidden_states", "PR", "SID", "ER", "ASR", "ASV", "SD", "QbE", "ST", "SS", "SE", "secret"]
      for task in tasks:
          # you can do task-specified pre- / post-processing depend on the arg "upstream_feature_selection"
          results = upstream(wavs, upstream_feature_selection=task)
          hidden_states = results["hidden_states"]
          assert isinstance(results, dict)
          assert isinstance(hidden_states, list)
  
          for state in hidden_states:
              assert isinstance(state, torch.Tensor)
              assert state.dim() == 3, "(batch_size, max_sequence_length_of_batch, hidden_size)"
              assert state.shape == hidden_states[0].shape
      ```
  
  #### get_downsample_rates
  
  Provide the downsample rate **from 16000 Hz waveforms** for each task's representation in the dict. For the standard 10ms stride representation, the downsample rate is 160.
  
  ```python
  SAMPLE_RATE = 16000
  MSEC_PER_SEC = 1000
  downsample_rate = SAMPLE_RATE * 10 / MSEC_PER_SEC  # 160
  ```
  
  The downsample rate will be used to:
  
  1. Calculate the valid representation length of each utterance in the output padded representation.
  2. Prepare the training materials according to the representation's downsample rate for frame-level tasks, e.g. SD, SE, and SS.
  
  - **Input:** the task key (str)
  - **Output:** the downsample rate (int) of the representation for that task
  
  ```python
  for task in tasks:
      assert isinstance(task, str)
      downsample_rate = upstream.get_downsample_rate(task)
      assert isinstance(downsample_rate, int)
      print("The upstream's representation for {task}"
          f" has the downsample rate of {downsample_rate}.")
  ```
  
  ### 2. Create an account and organization on the Hugging Face Hub
  
  First create an account on the Hugging Face Hub and you can sign up [here](https://huggingface.co/join) if you haven't already! Next, create a new organization and invite the SUPERB Hidden Set Committee to join. You will upload your model to a repository under this organization so that members inside it can access the model which is not publicly available.
  
  * [superb-hidden-set](https://huggingface.co/superb-hidden-set)
  
  ### 3. Create a template repository on your machine
  
  The next step is to create a template repository on your local machine that contains various files and a CLI to help you validate and submit your pretrained models. The Hugging Face Hub uses [Git Large File Storage (LFS)](https://git-lfs.github.com) to manage large files, so first install it if you don't have it already. For example, on macOS you can run:
  
  ```bash
  brew install git-lfs
  git lfs install
  `