Solved by ptrblck in post #2 
              
              
                Yes, that should be the case, which would create a zero gradient for the last linear layer: 
model = models.resnet18()
data = torch.randn(2, 3, 224 , 224)
target = torch.randn(2, 1000)
criterion = nn.MSELoss()

output = model(data)
output = output[:, :100]
target = target[:, :100]
loss = criterion(o…