Hi,
Running empty_cache at the beginning of your process is not useful as nothing is allocated yet.
When you restart the kernel, you force all memory to be deallocated.
So if you still run out of memory it is simply because your program requires more than what you have. You will most likely have to reduce the batch size or the size of your model.