Solved by iffiX in post #6 
              
              
                You can add this line: 
advantages_minib = advantages_minib.detach()

after: 
if self.normalize_adv:
    advantages_minib = (advantages_minib - advantages_minib.mean()) / (advantages_minib.std() + 1e-8)