Answers.Body: <p>Given your response in the comments, you are faced with a <a href="https://en.wikipedia.org/wiki/Semi-supervised_learning" rel="nofollow noreferrer">semi-supervised learning problem</a> where you have a small set of data with ground truth labels, and a large set of data without ground truth labels.</p>
  <p>If you look at a similar problem, predicting ImageNet label with only 10% of the ground truth labels, you can see that the best methods reach <a href="https://paperswithcode.com/sota/semi-supervised-image-classification-on-2" rel="nofollow noreferrer">~80% accuracy</a>.</p>
  <p>You can try to reproduce the approach of <a href="https://arxiv.org/pdf/2006.10029v2.pdf" rel="nofollow noreferrer">Big Self-Supervised Models Are Strong Semi-Supervised Learners</a>, which achieves ~74% accuracy on ImageNet with only 1% of the labels.</p>
  <p>Namely:</p>
  <ol>
  <li><strong>Unsupervised pre-training step</strong>. Train a deep and wide model on the unlabelled data. The idea is to minimize a contrastive loss using slightly modified versions of the training samples. The method they introduce is called SimCLRv2, which you can reuse in your problem by using data augmentation methods suited to audio samples. You can read on this topic in <a href="https://pytorch.org/tutorials/beginner/audio_data_augmentation_tutorial.html" rel="nofollow noreferrer">this Pytorch tutorial</a>.</li>
  <li><strong>Fine tuning step</strong>. Fine tune the model obtained in step 1 on the small dataset for which you have the ground truth labels. To do this, replace the last few layers of your big unsupervised model with an MLP and train it with a classic cross entropy loss (just the MLP part, the rest of the model should be frozen).</li>
  <li>[Optional] <strong>Distillation step</strong>. Use the model of step 2 to label the data for which you didn't have ground truth labels and train a smaller network in a supervised manner on these new labels (and only those, exclude the data for which you had the ground truth labels!).</li>
  </ol>
  <p>I've simplified a lot the method described in the paper, to give you an idea of the general approach. Read the paper for details!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-28T20:48:46.107Z
Answers.Text: Would you mind accepting my answer if it's satisfactory?
Answers.UserDisplayName: Raphael Lopez Kaufman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T18:51:08.823Z
Answers.LastActivityDate: 2022-07-27T18:51:08.823Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Answers.Body: <p>If I understand correctly, the training data are voice messages which contain one or more suggested labels for the class that the voice message belongs to. Even though there are multiple suggestions, each voice message only truly belongs to one class.</p>
  <p>If so, you could define the output layer of the network to have 10 nodes (the number of classes), with each node outputting values between 0 and 1 using a sigmoid function.</p>
  <p>The loss to base optimisation could be the MSE of the sum of the element-wise difference between the input vector say [1,1,1,0,0,0,0,0,0,0] (the first 3 classes are suggested) and the output vector of the network [0.78, 0.76, 0.65, 0.23 ... 0.11, 0.36].</p>
  <p>Once the model is sufficiently trained, for prediction you could apply a softmax to the output vector and select the largest value as the class to select.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-09T07:08:31.690Z
Answers.LastActivityDate: 2022-07-09T07:08:31.690Z
Answers.UserDisplayName: dmc-au
Body: <p>I have voice recordings which are labelled by not only a single label but multiple labels. Each voice recording corresponds to one of class labels within a set. In other words, the training instance is given a <strong>set</strong> of (or distribution over) candidate class labels and <strong>only one</strong> of the candidate labels is the correct one.</p>
  <p>I wish to train a model that classifies which class label corresponds to each voice recording. Each one of my voice recordings is accompanied by a set of 10 potential labels (labels are always different), but it is unknown which label it is exactly (aside from a small sample where there is only one correct label).</p>
  <p>This is due to the nature of where my data comes from: someone records a short voice message and then types the same message into a chat, however there will be slight delay between the two and in the meanwhile other chat messages arrive. Only one of the next 10 chat messages after the voice message is the correct one that corresponds to that voice message.</p>
  <p>How would I define a loss function in this case?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-07T19:54:17.210Z
Comments.Text: Do you have the ground truth label or just the set of 10 potential labels?
Comments.UserDisplayName: Raphael Lopez Kaufman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-08T09:05:34.237Z
Comments.Text: @RaphaelLopezKaufman Ground truth label is among the 10 potential labels
Comments.UserDisplayName: MilTom
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-10T22:30:07.670Z
Comments.Text: @MilTom " Ground truth label is among the 10 potential labels", but does this mean that you know what the ground-truth label (among these 10) is, or do you only know that the ground-truth label is among the 10 potential ones? It's important to clarify this, to understand if you can apply supervised learning or not.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-11T07:59:07.657Z
Comments.Text: @nbro I do not know what the ground-truth label is - I just know it is one of the 10 potential labels. I do have a small set that is one-to-one labelled, however most of the dataset is not.
Comments.UserDisplayName: MilTom
Tags: <recurrent-neural-networks><objective-functions><audio-processing><multi-label-classification><voice-recognition>
Title: How to define a loss function for multi-label problem?
UserDisplayName: MilTom
ThreadText: # How to define a loss function for multi-label problem?
  
  (Asked by: username_0 on 2022-07-07)
  
  I have voice recordings which are labelled by not only a single label but multiple labels. Each voice recording corresponds to one of class labels within a set. In other words, the training instance is given a **set** of (or distribution over) candidate class labels and **only one** of the candidate labels is the correct one.
  
  I wish to train a model that classifies which class label corresponds to each voice recording. Each one of my voice recordings is accompanied by a set of 10 potential labels (labels are always different), but it is unknown which label it is exactly (aside from a small sample where there is only one correct label).
  
  This is due to the nature of where my data comes from: someone records a short voice message and then types the same message into a chat, however there will be slight delay between the two and in the meanwhile other chat messages arrive. Only one of the next 10 chat messages after the voice message is the correct one that corresponds to that voice message.
  
  How would I define a loss function in this case?
  
  **Tags:** recurrent-neural-networks, objective-functions, audio-processing, multi-label-classification, voice-recognition
  > **Comments:**
  > **username_1**: Do you have the ground truth label or just the set of 10 potential labels?  
  > **username_0**: @RaphaelLopezKaufman Ground truth label is among the 10 potential labels  
  > **username_2**: @MilTom " Ground truth label is among the 10 potential labels", but does this mean that you know what the ground-truth label (among these 10) is, or do you only know that the ground-truth label is among the 10 potential ones? It's important to clarify this, to understand if you can apply supervised learning or not.  
  > **username_0**: @nbro I do not know what the ground-truth label is - I just know it is one of the 10 potential labels. I do have a small set that is one-to-one labelled, however most of the dataset is not.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-07-27. Score: 1 | Accepted answer
  
  Given your response in the comments, you are faced with a [semi-supervised learning problem](https://en.wikipedia.org/wiki/Semi-supervised_learning) where you have a small set of data with ground truth labels, and a large set of data without ground truth labels.
  
  If you look at a similar problem, predicting ImageNet label with only 10% of the ground truth labels, you can see that the best methods reach [~80% accuracy](https://paperswithcode.com/sota/semi-supervised-image-classification-on-2).
  
  You can try to reproduce the approach of [Big Self-Supervised Models Are Strong Semi-Supervised Learners](https://arxiv.org/pdf/2006.10029v2.pdf), which achieves ~74% accuracy on ImageNet with only 1% of the labels.
  
  Namely:
  
  1. **Unsupervised pre-training step**. Train a deep and wide model on the unlabelled data. The idea is to minimize a contrastive loss using slightly modified versions of the training samples. The method they introduce is called SimCLRv2, which you can reuse in your problem by using data augmentation methods suited to audio samples. You can read on this topic in [this Pytorch tutorial](https://pytorch.org/tutorials/beginner/audio_data_augmentation_tutorial.html).
  2. **Fine tuning step**. Fine tune the model obtained in step 1 on the small dataset for which you have the ground truth labels. To do this, replace the last few layers of your big unsupervised model with an MLP and train it with a classic cross entropy loss (just the MLP part, the rest of the model should be frozen).
  3. [Optional] **Distillation step**. Use the model of step 2 to label the data for which you didn't have ground truth labels and train a smaller network in a supervised manner on these new labels (and only those, exclude the data for which you had the ground truth labels!).
  
  I've simplified a lot the method described in the paper, to give you an idea of the general approach. Read the paper for details!
  
  > **Comments:**
  > **username_1**: Would you mind accepting my answer if it's satisfactory?  
  
  ---
  
  ### Answer by username_3 on 2022-07-09. Score: 1
  
  If I understand correctly, the training data are voice messages which contain one or more suggested labels for the class that the voice message belongs to. Even though there are multiple suggestions, each voice message only truly belongs to one class.
  
  If so, you could define the output layer of the network to have 10 nodes (the number of classes), with each node outputting values between 0 and 1 using a sigmoid function.
  
  The loss to base optimisation could be the MSE of the sum of the element-wise difference between the input vector say [1,1,1,0,0,0,0,0,0,0] (the first 3 classes are suggested) and the output vector of the network [0.78, 0.76, 0.65, 0.23 ... 0.11, 0.36].
  
  Once the model is sufficiently trained, for prediction you could apply a softmax to the output vector and select the largest value as the class to select.
  
  
  ---
  
  