apaszke:

A more precise mathematical definition would be to say that what we often refer to as “gradients” are Jacobian-vector products, and the gradient is the vector you want to multiply with it (for the loss it’s initialized as [1] by default, so it gives the Jacobian itself).


Hi,
I implement a skip-gram model using pytorch. However, I find the backward speed depends on the embedding size. The larger is the embedding size, the slower is the backward speed. So I’m wondering if the backward will calculate the gradient of each variable, even the variable that is not used in this step?
For example, I have an Embedding variable which is 10,000*100. When I used 10th and 20th embedding, will backward function calculate the gradient only for this two embeddings or will it calculate the gradient for all 10,000 embeddings?