I think there is a GPU memory leak. These are usually caused if you are not detaching your gradient information during inferring and saving your variables to a list. Can you post your source code here?