Understood, I got this working this morning.
However, I tried to make it work with resnet34 as backbone and it throws me Dilatation>1 not supported.
There is a pull request that solves this:

  
      github.com/pytorch/vision
  
  
    
  
    
  

  
    
      ResNet BasicBlock dilation
    

    
      pytorch:master ← christian-rauch:resnet_dilation
    

    
      
        opened 
        
          
        
        Apr 17, 2020
      
      

      
        
          
          christian-rauch
        
      

      
        
          +2
          -4