I think I have found the culprit. It is the line
input_normalized = (input - sample_mean.expand_as(input))/sample_var.expand_as(input)
in the forward call of BatchReNorm1d. But I don’t know why is this a problem? @jekbradbury  Please tell me what to do?
When I break this line into two lines:
input_normalized = (input - sample_mean.expand_as(input))
input_normalized = input_normalized/sample_var.expand_as(input)

things are working fine (at least I hope so). And this behaviour is intriguing.