Thanks for your reply! Your suggestions are really helpful. I deleted some variables that I generated during training for each batch. Currently, the memory usage would not increase but the training speed still gets slower batch-batch. I double checked the calculation of loss and I did not find anything that is accumulated from the previous batch. Do you know why it is still getting slower?
I also tried another test. For example, the average training speed for epoch 1 is 10s. After I trained this model for a few hours, the average training speed for epoch 10 was slow down to 40s. So I just stopped the training and loaded the learned parameters from epoch 10, and restart the training again from epoch 10. I though if there is anything related to accumulated memory which slows down the training, the restart training will help. However, after I restarted the training from epoch 10, the speed got even slower, now it increased to 50s per epoch.
Thanks!