Hey, thanks for the answer!
It was not, but I found this post by Soumith Chintala

  
    
    
    Why don't we put models in .train() or .eval() modes in DCGAN example 
  
  
    I’m trying to figure out, why don’t we put generator model in eval() mode at the end when using fixed_noise as input or when we are just training the discriminator. I believe the batch norm layers should behave differently evaluation mode. 


I tried training with eval mode but the model collapses to one particular image.  
Thank you
  


which does greatly improve the performance!