Yes I have read that post but still don’t know what’s causing the problem here. Is there any way to make it not update the parameters from the previous batch? I thought that since the previous batch params haven’t been used to calculate the current loss they shouldn’t be updated.
Update: to be more specific, I still want to use Adam or SGD with momentum, I just don’t want the parameters to update if they haven’t been used to calculate the loss for the current batch.