Could you add these memory stat prints into your training loop and see, if they come dangerously close to the limit?
The memory stats might not reflect the usage during training, if you already hit the OOM and thus potentially the cudnn error.