Somehow this doesn’t seem right:
        critic_optimizer.zero_grad()

        target_actions = target_actor(next_state_batch)
        y = reward_batch + gamma * target_critic([next_state_batch, target_actions])
        critic_model.eval()
        critic_value = critic_model([state_batch, action_batch])
        critic_model.train()
        critic_loss = torch.mean(torch.square(y - critic_value))

        critic_loss.backward()
        critic_optimizer.step()

Didn’t I just exclude batch norm parameters from training effectively getting rid of normalization? I mean this is part of the training routine so it seems logical to have the model in train() mode. I have read the original Batch Norm paper and it seems to confirm that.
Now I’m really confused.
Just for testing I entirely commented out batch normalization and the model is training well without it.
So the conclusion is, there is something wrong with my batch normalization and I still don’t know what…