Solved by stas in post #2 
              
              
                So the understanding that came from working on GPU RAM fragmentation diagnostics is that this won’t cause an actual fragmentation, since an average model will be 100MB+ and a whole bunch of whole gpu memory pages will be freed and re-used later through remapping of free pages. 
So the only issue her…