The idea behind transfer learning for vision tasks is to assume that convolution filters are more or less universally applicable to different tasks or different distributions. So you freeze everything related to convolutional part and keep training just fully connected part. VGG has multiple dense layers, you need to let them train too.