If the architecture has good prediction power, shouldn’t it also be good at predicting 90 degree rotated images? I mean, if I made the mistake of feeding WH instead of HW for the whole trainset and test set, shouldn’t it be able to extract the features anyway? Or is this “rotation” any different in 4D? I would love to conceptually understand this.
I will do then the operation in PyTorch using permute and change the default initialization to see if something improves.
Yes, they oddly use only one node at the end. (They have another deeper architecture with 4 layers but i face same issues with that one)
Last thing:  Sigmoid activation on the last layer plus cross entropy loss is equivalent to NLL plus Softmax, I am right?
Thanks!!