The problem comes from this paper: Decoupling “when to update” from “how to update”.
Regardless the detail of the paper, this implementation will resulting mini-batches has different sizes.
For example, if we set the mini-batch size to 128. Some of the batches could have only 10 or even zero samples.
If there’s zero sample in a batch, then we can just skip the update at this iteration.
However, if the sample in a batch is much samller, let’s say 10% of the 128, then we have to scale down the update of the gradient on this iteration.
The idea is that if there is a less important batch, it shouldn’t have the same impact as a full-size batch.
If we reduce the learning rate 10 time on this paticular batch, we can achieve this effect, and use the normal learning rate on the full-size batches.
But how can we achieve this in PyTorch?
I hope my description is clear. Thank you so much for any suggestions.