Assume your training code (e.g. Trainer) can be viewed as a class object that contains the network, optimizer and etc.
My workable solution is like this.
import gc
retry_time = 0
while retry_time < 3:
        try:
             trainer = Trainer() # Init the network and optimizer inside
             result = trainer.train()
             break # Still goto finally after break while
        except RuntimeError as e:
             retry_time += 1
             print('Runtime Error {}\nRun Again......{}/{}'.format(e, retry_time, 3))
             if retry_time == 3:
                 print('Give up!')
                 break  # Still goto finally after break while
        finally:
             # Handle CUDA OOM Error Safely
             del trainer
             gc.collect()
             torch.cuda.empty_cache()

You can monitor the memory usage after the last line of the finally block before the next trial to see if the memory are released as you expect.