albanD:

Hi,
Could you send me a full script that runs with all the sizes being the ones you use. Replacing your data with random tensors.
I guess two possible things here:

The graph expands accross iterations and thus the traversal for the backward becomes dead slow. I need the full running code to check that.
Give the simplicity of your graph and the fact that it’s mainly Embedding from what I see, you might be hitting a very bad worst case. Here again, try sparse=true for the embedding, it was made for that purpose.



Thank you very much!
Here is my code:
import torch.optim as optim
import torch
import torch.nn as nn
import numpy as np
import os
import time
if __name__ == '__main__':
    os.environ['CUDA_VISIBLE_DEVICES'] = '0'
    syn0 = torch.randn((2829,100),requires_grad=True,device='cuda')
    syn1 = torch.randn((2829,100),requires_grad=True,device='cuda')
    optimizer = optim.SGD([syn0, syn1], lr=0.025)
    Lossfunc = nn.BCELoss(reduction='sum').cuda()
    start1 = time.time()
    for index,_ in enumerate(range(40000)):
        word_in = np.random.randint(low=0, high=2829, size=32)
        word_out = np.random.randint(low=0, high=2829, size=32)
        if index%10000 == 0:
            print(('%d of 40000 (%.2f%%)')%(index,index / 400.0))
        word_in1 = torch.cuda.LongTensor(word_in)
        word_out1 = torch.cuda.LongTensor(word_out)
        label = torch.cuda.FloatTensor([1]+ [0]*31)
        emb_u = nn.functional.embedding(word_in1,syn0,sparse=True)
        emb_v = nn.functional.embedding(word_out1,syn1,sparse=True)
        outs = torch.sigmoid(torch.sum(torch.mul(emb_u, emb_v), dim=-1))
        loss = Lossfunc(outs,label)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print (time.time()-start1)

When i replace my data with random tensor,  the backward become about 2x slowly than the forward pass just like you said.
But it’s still very slow, it took about 1 min to run 40000 rounds in a Tesla M40. Is there any mistake in my code writing? If not, can you give me some suggestion to speed up it? Should i use torch.set_num_thread() or torch.multiprocessing?
Thanks again!