Try as suggested by @pchandrasekaran
Also, how’s your model behaving when you try to run it on CPU? In my personal experience running some models on GPU resulted in exploding loss sometimes (the reason for which is still unclear to me). Try running it on CPU once.