You should not use empty_cache() as it can only slow down your code.
If the GPU is used at 99%, then there isn’t much more you can do I’m afraid.
You can try the bottleneck 63 tool maybe to get more information.