Thank you again for your suggestions.



 ptrblck:

Could you flatten the tensors starting in dim1 ?


This doesn’t seem to make a difference in terms to the high loss value. However, I have tried changing the optimizer from SGD to Adam, and the loss values begin to drop (although still really high).



 ptrblck:

I would recommend to check some outputs and targets manually and make sure that the values are in the expected range and that no unwanted permutation was appiled somewhere in the code.


When I view the outputs and targets, they look like they’re within the expected range (I don’t see any large variations in the predicted keypoints when comparing to the ground truth values).



 ptrblck:

Your model might create large intermediate tensors, which would create this OOM issue. The number of input features is also relatively large in the linear layer, which would also point towards a high memory usage.


You were right, I adjusted my model by adding some more convolutional layers etc, and now the model is working.