Solved by User4 in post #5 
              
              
                Thank you, I try your solution but it is not work with the code above. 
However, I found what is the problem, I set the validation batch size is twice the training batch size, when I set them equal, the GPU memory is very stable.