erm:

If the architecture has good prediction power, shouldn’t it also be good at predicting 90 degree rotated images? I mean, if I made the mistake of feeding WH instead of HW for the whole trainset and test set, shouldn’t it be able to extract the features anyway? Or is this “rotation” any different in 4D? I would love to conceptually understand this.


I think it shouldn’t matter, if you are retraining the model and also flattening the tensor for this particular model.



 erm:

Sigmoid activation on the last layer plus cross entropy loss is equivalent to NLL plus Softmax, I am right?


No, nn.CrossEntropyLoss expects raw logits (no activation at the end), while nn.NLLLoss expects log probabilities (F.log_softmax at the end).
Since you are using the second approach, it should be fine.