But the following code based on numpy could run much faster for the same training process, just using CPU(4s VS 65s):
        syn0 = np.random.uniform(low=-0.5/100, high=0.5/100, size=(2829, 100))
        syn1 = np.zeros(shape=(2829,100))
        start1 = time.time()
        for index,_ in enumerate(range(40000)):
            x_ = np.random.randint(low=0, high=2829, size=32)
            if index%10000 == 0:
                print(('%d of 40000 (%.2f%%)')%(index,index / 400.0))
            context_word = np.random.randint(low=0, high=2829, size=1)[0]
            label = np.array([1] + [0]*5)
            z = np.dot(syn0[context_word], syn1[x_].T)
            p = expit(z)
            g = 0.025 * (label - p)
            neu1e = syn1[x_]
            syn1[x_] += np.outer(g, syn0[context_word])
            syn0[context_word] += np.dot(g, neu1e)
        print(time.time() - start1)

I want to use autograd function provided by pytorch, but it slow down severely.
I tried increase the batch size, and the performance will drop correspondingly.