Because I use the pretrain network, where the CNN layers will be frozen so the trained parameters would not change. I then replace the fully connected layer with my own custom layers.
All the process is refer to the official pytorch guide, when want to do transfer learning.