This should help. I had a similar question recently…

  
    
    
    Clarification on re-initializing optimizer in every epoch 
  
  
    You shouldn’t be doing that, right? fmi as well as my issue here