Hi @ptrblck, thanks for your help, I executed nvidia-smi on windows but I only got N/A for each process’ gpu usage, however, I do find the cause to my problem.
Since I load data from tfrecord file, I import tensorflow to do data preprocessing, and tf takes up all the gpu memory by default. I flush CUDA after the preprocessing and everything works fine now!