Assuming you are trying to classify your data into 4 classes, your model output should have the shape [batch_size, 4].
I’m not sure, how nn.ConvTranspose2d should deal with your activation volumes, as it’ll increase the spatial size in your setup.
Could you explain your architecture a bit?
I’m familiar with an adaptive pooling layer before the linear or with global pooling for fully convolutional models. However, I’m currently not sure how your models should works.