Thank you, I try your solution but it is not work with the code above.
However, I found what is the problem, I set the validation batch size is twice the training batch size, when I set them equal, the GPU memory is very stable.