Thanks @ptrblck. Data loading, processing, and each line in model have been tested. The bottle neck is the backward() part. It will increase a much larger than other mthod when I increase batch size for this model structure. Other models like pure LSTM, GRU and CNN will be fine, although the time increased for backward() in each time step but overall time will decrease.
I have some thinkings about the model strusture. I have used two View() in model to reshape the data. And all my structure with the view() will have this problem. So I think it may be due to this but I dont know why. I have also seen some of you previous answers on view() in forward function but I don’t know whether it is the problem.