Thanks @ptrblck for your reply.
I don’t have any other modules, which switch their behavior in train/eval and set  model.eval() in training phase. Now I am seeing Nan values in loss. However this is not the case when I set model.train() in my code.
Further I also tried to set individual batch norm layers to eval() mode (following your code snippet). Again I am observing Nan values.
Can you suggest anything that might be possibly going wrong?