Hi,
Some observations:

My first point does not happen, so no problem on that side.
Sparse=True make the perf slightly worst here. This is expected as your embedding is not that big.
In my tests, I moved all the data generation out of the loop but that does not change much.
Here each forward-backward-update takes <1ms on my machine. I don’t think you can expect it to be much faster. It’s just that the outer loop is very large. Try removing the python if statement in your loop, you actually see the difference in runtime.
The gpu usage is actually quite low, increasing the batch size to 128 still gives me a runtime of <1ms per iterations. So If you want this to run faster, increase the batch size.
torch.set_num_thread will only change cpu core usage for heavy operations. But you don’t do any such operation of cpu here.
torch.multiprocessing would allow you to do multiCPU-multiGPU. But you can’t use the one you have already fully, so there is little hope you can improve on that side.