What kind of model are you using?
Are you dealing with some dynamic sizes, e.g. varying batch shapes?
Also, did you make sure to write functions for the train and validation method?
Are you running other processes on this GPU, e.g. your desktop output, or are you using this GPU just for computation?