wwaayyaaww:

I found that the GPU memory allocated for evaluation couldn’t be cleared (about 4G memory being occupied as illustated in the figure).


This is expected, since the memory would be in the cache assuming that you have deleted all tensors. If not delete objects, which are not needed anymore.



 wwaayyaaww:

This causes the memory shortage issue during training even if I set the batch size to only 16.


I assume that you’ve verified this claim by running the training without a previous evaluation and this setup worked fine?