Turns out I used align_corners=True in the upsampling step. This adds a location dependent phase shift to the upsampled image relative to decoder arm. Enough for the network to create filters to exploit that apparently…  align_corners=False leaves it invariant to translations. Nice fig here:

  
    
    
    What we should use align_corners = False vision
  
  
    [align_corners] 
Here is a simple illustration I made showing how a 4x4 image is upsampled to 8x8. 
When align_corners=True, pixels are regarded as a grid of points. Points at the corners are aligned. 
When align_corners=False, pixels are regarded as 1x1 areas. Area boundaries, rather than their centers, are aligned.
  


So I guess it may in general be a bad idea to use align_corners=True if you want translational equivariance!