@ptrblck Thank you for your reply.
I had one more question regarding the topic following your answer:
Let’s suppose that my forward method looks like this:
def forward(self, x, hs):
          out, hs = self.lstm(x, hs)
          out = self.batchnormalization(out)
          out = self.fullyconnected(out)
          return out, hs

Then in my training and validation code:
for e in range(epochs):
        training_loss = 0
        validation_loss = 0
        model.train()
        for x,y training_data:
            train_h  = model.init_hidden(batch_size) 
            train_h = tuple([h.data for h in train_h])  
            out, train_h = model(x, train_h)
            **loss = criterion(out,y)** 
            training_loss += loss.item()
            opt.zero_grad()
            loss.backward()
            optimizer.step()
        training_losses.append(training_loss/len(training_data))

              # validation
                model.eval()
                with torch.no_grad():
                    for x, y in validation_data:
                        val_h  = model.init_hidden(batch_size) 
                        val_h = tuple([h.data for h in val_h])
                        out, val_h = model(x, val_h)
                        **loss = criterion(out, y)**
                        validation_loss += loss.item()
                validation_losses.append(validation_loss/len(validation_data))

Then in my training and validation code how should I use the running stats that you mentioned? Since as far as I understand, I cant just do loss = criterion(out, y), right? (because the ‘out’ was calculated using the BatchNorm)