Solved by ptrblck in post #2 
              
              
                You are most likely running into the JIT kernel compilation, since you are not using sm_80 or sm_86 in your binaries due to CUDA10.1. 
Use the CUDA11.0 binaries and the startup time should be gone.