Recently, I used the function torch.cuda.empty_cache() to empty the unused memory after processing each batch and it indeed works (save at least 50% memory compared to the code not using this function).
At the same time, the time cost does not increase too much and the current results (i.e., the evaluation scores on the testing dataset) are more or less OK.
Since I just do the comparison on my program, it is too limited to make a conclusion. I wonder if this function will affect the training of the model?