Could you try to see where it hangs, maybe add logging or see which GPUs are busy and idle during training?
You can also do an experiment where you run on 1 TITAN and 1 RTX3090 only and see if such training completes.
If the issue is some GPUs are faster and it leads to work imbalance as the result, maybe try to decrease batch size and see if it helps.