Hi ! I am trying to train a neural network composed of two “sub networks” which outputs each an embedding. I use the cosine similarity of these embeddings as the loss function (trying to maximize it for positive samples). Anyway, I do not understand why my loss.backward() is so slow.
Actually, the forward pass takes 0.008ms on average while the backward pass is about 0.9ms which is way too slow for such a small network (two Linear layers for the first part and only an Embedding layer for the second one). I can’t figure out what’s wrong.
In addition, if I interact with my python script with python debugger and then continue the training it happens that loss.backward speeds up from 0.9ms to 0.02ms. I do not understand this behavior either, maybe a memory access issue ?
Do you have any idea please?
Thank you in advance