hey everyone, you’ll probably hate me for dropping the ball on this. Sharding itself was harder than expected, I never finished it. Sharding for one layer was easy, but as you went across multiple layers, we need special data structures to deal with borders correctly.
Alternatively, here’s an approach that works well to fit large image based models and train them effectively:

  
      github.com/pytorch/pytorch
  
  
      
    
  


  Autograd container for trading compute for memory 103



  by prigoyal
  on 11:23PM - 10 Jan 18



  1 commits
  changed 1 files
  with 81 additions
  and 0 deletions.