Thanks for your reply! The memory reduces especially for the first batch after the cudnn benchmarking is commented. But the running time takes 30% longer.
From the output I posted above, I assume that the memory used by benchmarking algorithms can exceed 1.5GB. For example, if 15 GB out of 16GB is occupied by the model training itself, will this get an OOM error? I just wonder is there a way to so something like making cudnn benchmarking try these algorithms before the training to reduce the peak memory allocated?