Hi,
Google published a paper where they “pipeline” the inference (and backprop) stage (each later at a time).

  
      
      arXiv.org
  
  
    

Sideways: Depth-Parallel Training of Video Models 3

We propose Sideways, an approximate backpropagation scheme for training video
models. In standard backpropagation, the gradients and activations at every
computation step through the model are temporally synchronized. The forward
activations need to...


  
  
    
    
  
  


I thought that most frameworks do this during the forward pass when inferencing, since you don’t need to wait for the batch to finish calculating all the outputs before loading a new batch.
If that’s not the case, is any work on this planned for pytorch?
Thanks!