In general sampling with or without replacement It Is still quite an open topic for neural network with SGD or other optimizers
See also:
  

      stats.stackexchange.com
  

  
      
    
  


  Should training samples randomly drawn for mini-batch training neural nets be drawn without replacement? 3



  machine-learning, neural-networks, optimization, deep-learning



  asked by
  
  
    bobo
  
  on 10:53PM - 19 Sep 16 UTC


  

  
    
    
  

  



  

      stats.stackexchange.com
  

  
      
    
  


  Why do neural network researchers care about epochs? 4



  neural-networks, deep-learning, gradient-descent



  asked by
  
  
    Sycorax
  
  on 02:44AM - 24 Oct 16 UTC