Could you post an executable code snippet, which raises the OOM issue on this device, please?
The memory peak might increase e.g. if cudnn uses its benchmarking to profile different kernels.
However, this should not raise an OOM error, but skip the algorithm and clear the cache afterwards.