It the decrease in GPU utilization observable after an approx. fixed number of iterations or does your code slow down consistently?
Are you storing some variables somewhere, e.g. losses.append(loss)?
Do you see an increased usage in GPU or CPU memory?