I am experiencing very slow training on RTX 2070 on Windows 10 and the GPU is hardly used, while the same exact code is super fast on GTX 1080Ti. After some research I concluded that training must only be running on CUDA cores and not on Tensor cores. I tried using .half() for both model and for inputs, but the training is still just as slow. Even after I switched to using apex (I used the conda distribution from  here: https://anaconda.org/conda-forge/nvidia-apex 2) the training is still slow.
I am kind of out of options now.
Is there an example script using .half() or apex that I can use to check whether GPU can be fully used?
Is there a way to monitor whether Tensor cores are being used on Windows?
Any help is appreciated!