Solved by ptrblck in post #2 
              
              
                You should do it once before the actual training starts, as the memory would be pre-allocated and moved to the cache afterwards. As long as you don’t clear the cache via torch.cuda.empty_cache() you wouldn’t have to rerun it.