Solved by ptrblck in post #4 
              
              
                The model itself seems to be correct, as I can perfectly overfit a random input and target batch of 10 samples: 
class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3,64,3,2,1)
        self.conv2 = nn.Conv2d(64,128,3,2,1)
        self.conv3 = nn.…