There are a few implementation of the first one available: https://github.com/daviddao/pytorch-neural-search-optimizer 6 or https://github.com/HaraldKorneliussen/pytorch-neural-search-optimizer 1
However, I have not found (yet?) an implementation for the second one (drop).