What I figured out right know is that my costumised resnext50 decides for the complete classification only to choose the class with the most observations. This means if I would train the network on three classes A, B and C and the test set contains class A (30 observations), B (20 observations) and C (50 observations), the results would have an accuracy of 50%, since the network alaways decides to go for class C.