Lower your batch size maybe? I ran a forward pass on your model here on my 2080 (8GB) with a batch size of 16. Since you have a Turing GPU, you could also use mixed precision training to reduce your model’s footprint.