Solved by ptrblck in post #5 
              
              
                Thanks for the model definition! Could you post the input shapes you are using as well, please? 

Your local CUDA toolkit and cudnn library won’t be used, if you’ve installed the conda binaries or the pip wheels. 
You could indeed build from source and compare the speed. If you are seeing a speedup …