I don’t know what you mean by “CUDA and cudnn mismatch”, but you could try to use e.g. the CUDA10.2 binaries and see, if you would get a similar performance.
Also note, that your training might suffer other bottlenecks such as the data loading, which should be visible by a low GPU utilization in nvidia-smi.