Let me give you all of my possible permutation of networks. 
Rule1: pre-trained net: is a network without any training, just loaded the the pre-trained weights and initialized the extra layers.
Rule2: checkpoint: is a network initialized by the pre-trained weight and learnt on my dataset (no parameter updating on my pre-trained section - frozen part).

pre-trained and pre-trained in training mode: the outputs are equal.
pre-trained and pre-trained in evaluation mode: the outputs are equal.
pre-trained and checkpoint0 in training mode: the outputs are equal.
pre-trained and checkpoint0 in evaluation mode: the outputs are not equal.
checkpoint0 and checkpoint1 in training mode: the outputs are equal.
checkpoint0 and checkpoint1 in evaluation mode: the outputs are equal.

I did not have any special layer in my net. All of them are the regular layers such as Conv2d, Maxpool, BatchNorm and nothing else.