Looks fine to me!
Just wondering why did you need to convert BGR to RGB? Did you load the images with OpenCV or is there another reason?
Back to the original issue:
You are loading the pre-trained VGG model and would like to fine-tune it on the ImageNet dataset?
Once it’s loaded, could you set it to eval and calculate the accuracy on the test data before any fine-tuning?
The error might also be related to a large learning rate. Since your model is already in a good state, a learning rate of 1e-3 might catapult your weights in a bad zone.