empty_cache will only release the unused and cached memory.
If the parameters and activations need more memory, you will run into an OOM error nevertheless.
You could try to lower the batch size or alternatively use torch.utils.checkpoint 4 to trade compute for memory.