I’m curious to hear the developers thoughts on Gradient Checkpointing.

  
      
      Medium – 14 Jan 18
  
  
    

Fitting larger networks into memory. – Yaroslav Bulatov – Medium 92

TLDR; we release the python/Tensorflow package openai/gradient-checkpointing, that lets you fit 10x larger neural nets into memory at the…

  Reading time: 12 min read
    

  
  
    
    
  
  


Memory limitations are one of the biggest restrictions I encounter both with pytorch and with deep learning in general and this seems like an interesting and possibly fruitful solution, particularly if it were baked right into the library itself.