My program has various sized inputs, sometimes OOM occurs, I used the above technique, like @marcel1991’s pseudocode, and continue to another iteration if OOM occurs, it seems that the  torch.cuda.empty_cache() is not enough to clean up the memory, and I could actually not continue to train – every iteration after that suffers from OOM, how can I clean up the allocated GPU memory when OOM occurs?