No, the forward is already defined in the model, I didn’t change anything in the model. It’s a simple code. Basically, all I have is a data-loader, the model, train and test.
I need to mention another warning
"    “/home/nimrod/anaconda3/lib/python3.6/site-packages/torchvision/models/densenet.py”, line 222, in forward
out = F.avg_pool2d(out, kernel_size=7, stride=1).view(features.size(0), -1)"
Maybe it’s all because the images are too small. I see this warning only with dense121.
On ResNet18 I get:
"
File “/home/nimrod/anaconda3/lib/python3.6/site-packages/torch/nn/modules/pooling.py”, line 547, in forward
self.padding, self.ceil_mode, self.count_include_pad)"
It leads me to believe that maybe the input/output size runtime error I had with resnet18/50, and densenet121 is because I trained them on 3232 images. After all, I use networks that were trained on Imagenet with kernel sizes  that might be bigger than some layer’s output. On Udacity they use cats vs. dogs, on pytoch you have heymenoptera, both are not too small. I ran the heymenoptera’s code (all of it)  from the pytorch tutorial and it worked well. And you  used 224224 input.
I think that if I want to do transfer learning on small images then I need to reduce the kernel size (and check padding) of such networks, and it makes sense that it requires some tweaking. It requires a close look at the network I work with. That’s probably the reason why I had that problem.
It looks like I wanted to learn how to do TL the quickest way (load cifar10), but learned a good lesson