Solved by spanev in post #2 
              
              
                Hi @Rojin 
I believe this comes from the fact that the KL divergence is an integral/sum. 
So the sum reduction would be the more paper faithful approach, but having the PyTorch default (mean) will still work (only have downscaled gradients).