Most major difference between my model and nvidia’s is this modification.

  
      github.com
  
  
    bfs18/tacotron2/blob/master/train.py#L251

x, y = model.parse_batch(batch)
y_pred = model(x)


loss = criterion(y_pred, y)
if model.mi is not None:
    # transpose to [b, T, dim]
    decoder_outputs = y_pred[0].transpose(2, 1)
    ctc_text, ctc_text_lengths, aco_lengths = x[-2], x[-1], x[4]
    taco_loss = loss
    mi_loss = model.mi(decoder_outputs, ctc_text, aco_lengths, ctc_text_lengths)
    if hparams.use_gaf:
        if i % gradient_adaptive_factor.UPDATE_GAF_EVERY_N_STEP == 0:
            safe_loss = 0. * sum([x.sum() for x in model.parameters()])
            gaf = gradient_adaptive_factor.calc_grad_adapt_factor(
                taco_loss + safe_loss, mi_loss + safe_loss, model.parameters(), optimizer)
            gaf = min(gaf, hparams.max_gaf)
    else:
        gaf = 1.0
    loss = loss + gaf * mi_loss
else:
    taco_loss = loss



  
  
    
    
  
  


Without gradient balancing lost curve much better (red)
                    
            
           1


Does it matter if loss calculated in forward or outside?
Also is it good idea in general to balance loss value like this?
Tacotron by its own have 2 losses

  
      github.com
  
  
    bfs18/tacotron2/blob/master/loss_function.py#L19

mel_target, gate_target = targets[0], targets[1]
mel_target.requires_grad = False
gate_target.requires_grad = False
gate_target = gate_target.view(-1, 1)


_, mel_out, mel_out_postnet, gate_out, _ = model_output
gate_out = gate_out.view(-1, 1)
mel_loss = nn.MSELoss()(mel_out, mel_target) + \
    nn.MSELoss()(mel_out_postnet, mel_target)
gate_loss = nn.BCEWithLogitsLoss()(gate_out, gate_target)
return mel_loss + gate_loss



  
  
    
    
  
  


Do you think it should be balanced too?
About batchnorm thing, i don’t understand how batchnorm affect params.
I mean it change only existed variables, do not freeze layers or something.
It work with batchnorm layers and raise error without.
                    
            
           1


May be because without batchnorm i have only one layer in module?
Or it doesnt matter?
I made another thread about memory usage (it is not only distributed thing) with example, hope it helps.

  
    
    
    + 50% VRAM used on torch 1.3 compared to 1.2 
  
  
    1.3 
https://colab.research.google.com/drive/104LtQ1zIioIOMQEPgVve77m5Rd4Gm0wU 
1.2 
https://colab.research.google.com/drive/1y4LF1a90PYfKvFgQw6fCMYKtIFK07mVr 
Everything same except torch version. 
At the end nvidia-smi show 11gb and 7 gb memory used.