I meant, you can’t parallel the sequential step in a LSTM (source code reference that OP cited), because it’s … well sequential. I’m sure torch.nn.LSTM is also doing a for loop inside
But like I said, if it’s 2 separate LSTM, you can just simply do it on 2 threads or something.
Note that I didn’t comment on that batching thing. Now that I glance at it. I realize if you guys are talking about running the same LSTM model, then sure you should always process a batch of dozens of sentences or something, same as how you deal with images. Nobody runs LSTM on 1 or 2 sentences at a time. Also note that the gain here is because larger matrix computation is more efficient, it’s likely still using 1 CPU core (unless torch is using some library to auto-parallel matrix multiplication on CPU)