Thanks for the pointer. I had the confusion as I was using SparseEmbeddings with Pytorch 0.3.0. There the embedding corresponding to padding index was also updated. The bug was fixed in 0.3.1. Adding it here in case someone else stumbles across it https://github.com/pytorch/pytorch/issues/3506 22