title: Noisy training loss
body: <p>I am training encoder-decoder attention-based model, with batch size 8. I don't suspect too much noise in the dataset, however the examples come from a few different distributions.</p>
  
  <p>I can see a lot of noise in the train loss curve. After averaging (.99), the tendency is fine. Also the accuracy of the model is not bad. </p>
  
  <p>I'd like to understand what could be the reason of such shape of loss curve</p>
  
  <p><a href="https://i.stack.imgur.com/fYSR1.png" rel="noreferrer"><img src="https://i.stack.imgur.com/fYSR1.png" alt="noisy train loss"></a>
  <a href="https://i.stack.imgur.com/QqqIa.png" rel="noreferrer"><img src="https://i.stack.imgur.com/QqqIa.png" alt="averaged train loss"></a></p>