Not totally. But I warped the loss into the network to alleviate this issue to some extent, look at this link:

  
      github.com
  
  
    jialee93/Improved-Body-Parts/blob/e9868bbaed2114022bb3094d4a4fa220932e98e3/train_parallel.py#L145 2



# target tensor shape: [8,512,512,3], [8, 1, 128,128], [8,43,128,128], [8,36,128,128], [8,36,128,128]
images, mask_misses, heatmaps = target_tuple  # , offsets, mask_offsets
# images = Variable(images)
# loc_targets = Variable(loc_targets)
# conf_targets = Variable(conf_targets)


optimizer.zero_grad()  # zero the gradient buff


loss_ngpu = posenet(target_tuple)  # reduce losses of all GPUs on cuda 0
loss = torch.sum(loss_ngpu) / opt.batch_size
# print(loc_preds.requires_grad)
# print(conf_preds.requires_grad)
if loss.item() > 1e6:
    print("\nLoss is abnormal, drop this batch !")
    loss.zero_()
    continue
# print(loss.requires_grad)
loss.backward()  # retain_graph=True
# torch.nn.utils.clip_grad_norm(posenet.parameters(), args.max_grad_norm)
optimizer.step()  # TODO：可以使用累加的loss变相增大batch size，但对于bn层需要减少默认的momentum



  
  
    
    
  
  


Alternatively, I recommend to use distributed training.