I guess “cudnn taking more GPU memory” is because cudnn needs workspaces, and if we set cudnn.benchmark=True, it will allocate the max workspace for each algorithms, which will cost more mems. I have this trouble now which often gets OOM error using cudnn.