The problem was that the second last layer from ResNet50 gave a 4d vector in the shape of [batchsize, 2048,1,1] so after squeezing with torch.squeeze(x) it had the right dimensions to go into the MLP.