Hello!
I’m working on a project where we are building a simple image classifier that we intend to deploy on a video stream. Since the images from the video are quite custom-looking we are mostly using our own datasets, and this is where the problem starts.
The labels for our datasets are generated via an automatic process with some added manual oversight. The resulting dataset contains a small amount of dirty data (unclear images where a part/nothing of the class-object is in frame). These images are hard to classify for humans, and might not be suitable to train on. Unfortunately, there is no easy way to remove these images from the dataset so I’m looking for an alternative way to handle them.
Since this dirty data is relatively rare (say 5%), I was hoping to reduce its impact on training by identifying these samples at runtime and limit the loss they produce. I’ve come up with two solutions.
Solution 1
The assumption is that the dirty data would show up as samples with high loss, and this code would limit their loss.
loss_fn = nn.CrossEntropyLoss(reduction="none")
loss = loss_fn(outputs, labels)

loss_threshold = loss.median() * 5 # Hyperparameter
loss_scales = torch.ones_like(loss)
for loss_idx, loss_val in enumerate(loss):
    if loss_val > loss_threshold:
        loss_scales[loss_idx] = loss_threshold / loss[loss_idx]

loss = loss * loss_scales # Ensures no sample-loss is greater than loss_threshold
loss = loss.mean()

Solution 2
Same idea as solution 1, but using PyTorch’s clamp function.
loss_fn = nn.CrossEntropyLoss(reduction="none")
loss = loss_fn(outputs, labels)

loss_threshold = loss.median() * 5 # Hyperparameter
loss = loss.clamp(max=loss_threshold.item())
loss = loss.mean()


It is not obvious to me whether these solutions are good ideas to try. For example, I don’t quite understand what happens to the gradients when you limit/clamp the loss as in these solutions (some info here torch.clamp kills gradients at the border · Issue #7002 · pytorch/pytorch · GitHub).
Based on a few experiments, these solutions perform worse than baseline. One thing that actually work is to limit the gradients via the pytorch function torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=0.1)
Any suggestions to solve my problem would be appreciated 
Edit: Found some other threads that talk about this. No conclusive answer yet.
  
    
    
    Per-class and per-sample weighting 
  
  
    Thanks. I used above code and it got error in backward() 
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-22-88aafe38e0f1> in <module>()
     15 loss =(loss * sample_weight / sample_weight.sum()).sum()
     16 print (sample_weight.shape, loss.shape)
---> 17 loss.mean().backward()
     18 
     19 #loss_total = torch.mean(loss * weights)

1 frames
/usr/local/lib/python3.6/dist-p…
  


  
    
    
    How to weight the loss? 
  
  
    ...
for step, input in dataloader:
    outptut = model(input)  
    # top_prob, _ = torch.topk(F.softmax(output, dim=1)[0], 1)
    loss = criterion(output, label)


I want to weight the highest probability of softmax to the loss. 
ex) 
If, prob of softmax : 0.75 ->  Effect of this sample * 0.75 
If, prob of softmax : 0.12 ->  Effect of this sample * 0.12 
But the top_prob is the list (length == batch size) 
like top_prob = [0.72, 0.75, 0.82, ... 0.94] 
and loss is one scalar value. 
(It means th…
  


  

      stats.stackexchange.com
  

  
      
    
  


  What's the effect of scaling a loss function in deep learning? 1



  deep-learning, optimization, loss-functions



  asked by
  
  
    Chris
  
  on 09:10AM - 15 May 18 UTC