Solved by PistonY in post #2 
              
              
                I think this may caused by your input is too small, both vgg models have 2^5 downsample then pass by a nn.AdaptiveAvgPool2d((7, 7)) that’s mean your input (h,w) at least is 7*2^5 = 224.