I don’t know how well models pretrained on ImageNet would generalize to CIFAR10. Given that the resolution of the images is different, I would expect to see some gaps. Also note that you are not normalizing the images to a zero mean and a unit variance (which is done during the ImageNet pretraining), so you might want to add it and see if this could improve the results.