Dear PyTorch Community
I’m trying to implement this segmentation model into PyTorch code base:

  
      
      GitHub
  
  
    

mapillary/inplace_abn 3

In-Place Activated BatchNorm for Memory-Optimized Training of DNNs - mapillary/inplace_abn


  
  
    
    
  
  


Problem: it’s taking around 6 hours to train a single epoch on Mapillary Vistas from scratch (on a single GTX 1080). Is this normal?
My feeling is that it shouldn’t take this long as other models (e.g. u-net) take under an hour (about 15min). It also seems to lag on sending tensors to GPU if I use normal BN. If I use InPlaceABN instead of lagging on sending to GPU it does so on the backward call. Either way, it also seems to lag on printing the loss or using .item() on loss.
Can anyone suggest where it might be going wrong or how I could figure this kind of problem out? I’ve tried using Mapillary’s SingleGPU class but it didn’t seem to help…