A proper weight initialization might avoid the vanishing gradients.
Also, normalization layers will help (which you are using already).
Are you using pure PyTorch code in your model?
Can you make sure you are not detaching some tensors?
Also, a small code snippet might help, if that’s possible.