A popular technique that claims to reduce model RAM requirements is gradient accumulation. However, from what I could gather, it seems that it cannot be used in real world applications as it is incompatible with batchnorm.
See the discussion here:

  
      
      Deep Learning Course Forums – 22 Dec 18
  
  
    

Accumulating Gradients 13

I have a custom batch sampler that I am training with but I also want to accumulate the gradients and do weight updates only at the end of each epoch but not each batch. What would be the most fastai way to do this with minimal effort. Thanks !

  Reading time: 20 mins 🕑
    Likes: 132 ❤

  
  
    
    
  
  


My question is, are there are any recent proven workarounds or fixes that have been used in production environments that would let one use gradient accumulation?