You can use nn.AdaptiveAvgPool2d to reduce the spatial dimensions to 1x1(HxW) and then use 1x1 convolution to bring num_filters = num_classes. Use this after the last conv output from vgg.
self.classifier = nn.Sequential(
          nn.AdaptiveAvgPool2d((1, 1)), 
          nn.Conv2d(in_channels=512, out_channels=n_classes, kernel_size=1),
          nn.ReLU(inplace=True)
         )

Given what you have done above, I am not sure how you were able to calculate loss since the output will be num_classes, H//32, W//32. However one important thing to note is, do not use nn.Softmax at the end of the network when using nn.CrossEntropy(it combines softmax and nll loss). That can cause problems.