You were right, synchronization was the problem. For this reason my proposed solution didn’t work. I guess there’s really no way of getting around this overhead since my code checks the condition to decide whether to break out of a computation for-loop. Maybe I’m wrong?
for t in range(10)
            self.RNN(stuff)
            if my_tensor.float().sum().data[0] == 0:
                break

By the way, where can I read more about how the async stuff works in Pytorch? A quick google didn’t bring useful stuff up.