The first question I would ask is the number of GPU cores I have.
For instance if I have 8 cores, then torch.cuda.empty_cache() should be rephrased as something like this:
how_many_gpus = torch.cuda.device_count()
    for _ in range(how_many_gpus):
        torch.cuda.set_device(_)
        torch.cuda.empty_cache()

What have you tried so far?