Hi @ptrblck ,
I was wondering if you finally found a workaround to this problem. I am doing some Transfer Learning where I initialize a model with the parameters of another previously trained model and freeze the few last fully connected layers. Here is how I typically do it:
for name, param in self.model.named_parameters():
	#tells whether we want to use gradients for a given parameter 
	if freeze:
	    param.requires_grad = False
	    print("Freezing parameter "+name)

Transfer looks fine and parameters are initially identical, but when I compare the respective min,max,mean and std values for each parameter of each layer in both models,  some of the frozen instances start to vary after a few epochs. See below a case where I froze group1-4:
The model I am transferring the weights from:
Name                            Min          Max          Mean           Std  
----------------------  -----------  -----------  ------------  ------------  
module.group1.0.weight  -0.13601      0.135853     0.000328239    0.078537    
module.group1.0.bias    -0.129506     0.13031     -0.000709442    0.0761818   
module.group2.0.weight  -0.0156249    0.015625    -6.8284e-06     0.00902701  
module.group2.0.bias    -0.0150966    0.0152359    0.000233364    0.00887235  
module.group3.0.weight  -0.0110485    0.0110485    8.25103e-06    0.00637962  
module.group3.0.bias    -0.0109931    0.0109642   -0.000212902    0.00620885  
module.group4.0.weight  -0.0078125    0.0078125   -1.07069e-06    0.00451099  
module.group4.0.bias    -0.0077329    0.00775102  -0.000157763    0.00451984  
module.fc1.0.weight     -0.00195312   0.00195312  -1.05901e-08    0.00112767  
module.fc1.0.bias       -0.00195279   0.0019521    5.93193e-05    0.00113513  
module.fc2.0.weight     -0.0312486    0.0312499   -2.94543e-05    0.0180225   
module.fc2.0.bias       -0.0312394    0.0289709   -0.00238465     0.0186226   
module.fc3.0.weight     -0.100976     0.0989116   -0.00164936     0.0606025   
module.fc3.0.bias       -0.059265    -0.059265    -0.059265     nan           


The model that I initialized thtough TL:
Name                            Min          Max          Mean           Std  
----------------------  -----------  -----------  ------------  ------------  
module.group1.0.weight  -0.136078     0.136051     0.00138295     0.0788667   
module.group1.0.bias    -0.135537     0.135878     0.00912299     0.0691942   
module.group2.0.weight  -0.0156247    0.0156249   -2.81046e-05    0.00902321  
module.group2.0.bias    -0.0151269    0.0152803    0.000945539    0.0088397   
module.group3.0.weight  -0.0110485    0.0110485   -7.81598e-06    0.00637801  
module.group3.0.bias    -0.0110323    0.0109976   -0.000282283    0.00675859  
module.group4.0.weight  -0.0078125    0.0078125   -8.4189e-07     0.00451147  
module.group4.0.bias    -0.00777942   0.00779467  -2.26952e-05    0.00466924  
module.fc1.0.weight     -0.00195312   0.00195312   1.48078e-07    0.00112768  
module.fc1.0.bias       -0.00194499   0.00195289   5.32243e-05    0.00112104  
module.fc2.0.weight     -0.0312488    0.0312494   -5.54657e-06    0.0180232   
module.fc2.0.bias       -0.0304042    0.0306912    0.00134896     0.018436    
module.fc3.0.weight     -0.0996469    0.101409    -0.00436459     0.0568807   
module.fc3.0.bias       -0.0561954   -0.0561954   -0.0561954    nan     

Any insight would be appreciated!
Thanks,