I was getting the same error while using resent with input image size other than 224X224. Pretrained resnet model has a FC layer of size 512. So associated pre-trained weight for this full connected layer is of size 512X1000.  When we use existing implementation of resenet then it perform Avg pooling with kernel size 7X7 for generating input for this FC layer.
(torchvision/models/resnet.py line number 110)
        self.avgpool = nn.AvgPool2d(7, stride=1) #input size : 512X7X7 output size 512X1
        self.fc = nn.Linear(512 * block.expansion, num_classes) 
        # Perform X.W+b, here weight matrix is size of 512XNum_Classes

Since in my case input for the avgpool layer was not of size 512X7X7 (it was size of 512XNXM) hence output of avgpool layer was not 512X1 which result into failure in the matrix multiplication.
One way to fix it is using adaptive avg pooling layer which perform avg pooling in such a way that it produces any pre defined output size. As we are interested in getting output of size 1 so
self.avgpool = nn.AvgPool2d(7, stride=1) can be replaced with self.avgpool = nn.AdaptiveAvgPool2d(1).
There is all ready a PR request for it https://github.com/pytorch/vision/pull/155 99 but it is still in process.