Note that the empty_cache() is not really needed as if you are about to run out of memory, it will be called automatically. And calling it too much is going to slow down your code quite a bit.
How do you measure if the issue is “resolved” or not?
Is there a small code sample (30-40 lines) you could share to see if we can reproduce it?