Hi, somebody has already implemented adaptive softmax in pytorch, MIT license, see: https://github.com/rosinality/adaptive-softmax-pytorch 181
There’s a bug there, see comment here: https://gist.github.com/rosinality/0cdd8d6adb8463961f50bd1845faddf8#gistcomment-2272578 67
I’m currently running a language model experiment with this, and it seems to be working, but don’t know enough to fully ascertain the quality of this implementation.