@mrshenli
I am exactly using the same code. I couldn’t replicate similar results.
Correct me if I have misunderstood the concept of running two micro-batches concurrently,
 for s_next in splits:
            # A. s_prev runs on cuda:1
            s_prev = self.seq2(s_prev)
            ret.append(self.fc(s_prev.view(s_prev.size(0), -1)))

            # B. s_next runs on cuda:0, which can run concurrently with A
            s_prev = self.seq1(s_next).to('cuda:1') 

Here part A, and part B as shown in the comments must run concurrently to get the pipeline performance. Am I correct/wrong?
If so, as Pytorch eagerly executes the layers, it is not asynchronous? Am I following this right?
If both these clauses are true, having threads is required? Isn’t it?
My reasoning comes with the usage of the for loop. Here within the loop a concurrent execution could
happen if we use threads. Am I following this wrong?