Hi @albanD, thanks a lot for replying!
I attached two photos showing the gradients of the embedding layer once without detaching any embedding and once after detaching the pos embeddings. I also fixed the seed and the model (both are at same epoch and optimizer state). The model is trained sufficiently long at the time of showing the gradients. The gradients are not equal but non-zero, which is expected.
What I do not understand is that adding up the gradients (the link is above) gives zero in theory. This means the network will not update its parameters. However, this is luckily not the case. Now I am simply confused how triplet loss is actually optimized given my implementation. (Ps. I might have overlooked a detail in the theory part as well.)
posOff1328×392 14.2 KB allOn1208×390 14.4 KB