I am sorry if that was confusing. The code never gives 0 gradient at any case. But I am wondering how exactly PyTorch gets the gradients.
Isn’t it that when I do loss.backward(), the gradients will be accumulated over the base, pos and neg instances? If so, the weights at the embedding layer are actually the same tensors, thus I expect they will be updated by the sum of gradients (times const). Surprisingly, in theory, this sum turns out to be always zero, as shown in the figure. But since I do not get zeros, there must be something that I do not understand.
Ltrip_grad1434×644 40.8 KB