The number of input features of your self.fc is wrong. Currently, the activation before this layer has a shape of [batch_size, 64, 7, 7]. Either you set in_features=64*7*7 for self.fc or use an adaptive pooling layer with output_size=1 before the linear layer.