Solved by _singh in post #3 
              
              
                We found the reason of this slight increase. The state variable used for saving the checkpoint as above was not nullified during training epochs. 
Adding state = None after saving checkpoint helped. 
Issue observed only with models with batchnorm and potentially has to do with the buffers as, the sa…