I ran the code with model.eval(). After the last batch norm layer all the values are becoming Nan.However when I set model.train() the code is working fine.
Further I saw, the weights and bias parameters of the Batch Norm layers are going to Nan. This seems quite a weird behaviour.
At the following threads also people are trying to do the exact same thing. I followed all the solutions mentioned there but still no luck.

  
    
    
    Proper way of fixing batchnorm layers during training 
  
  
    @maunz 
I had same problem. It seems I setting eval mode wrong. 
This thread helped me Freeze BatchNorm layer lead to NaN 
def set_bn_to_eval(m): 
classname = m.class.name 
if classname.find(‘BatchNorm’) != -1: 
m.eval() 
net.apply(set_bn_to_eval) 
if you want to set_bn_to_eval of some subnet or some basenetwork then just use. 
net..apply(set_bn_to_eval) 
hope this helps.
  



  
    
    
    Freeze BatchNorm layer lead to NaN 
  
  
    Hi, everyone 
I want to freeze BatchNorm while fine-tuning my resnet (I mean, use global mean/std and freeze weight and bias in BN), but the loss is so large and become nan at last: 
iter =  0 of 20000 completed, loss =  [ 15156.56640625]
iter =  1 of 20000 completed, loss =  [ nan]
iter =  2 of 20000 completed, loss =  [ nan]

the code I used to freeze BatchNorm is: 
def freeze_bn(model):
    for name, module in model.named_children():
        if isinstance(module, nn.BatchNorm2d):
            …