Solved by Paarulakan in post #3 
              
              
                I just made this change - zeroing grad right after optimizer.step() and this works. I cannot figure out why, but this works. 
          for j in tqdm(range(self.train_feed.num_batch), desc='Trainer.{}'.format(self.name())):
                self.optimizer.zero_grad()
                input_ = self.tra…