The long startup time is most likely create due to a JIT compilation of the CUDA code, if your installed PyTorch version wasn’t built for compute capability 8.0 (A100).
This would be the case, if you’ve installed the 1.4 binary instead of building it from source.
We are working towards building the PyTorch nightly binaries with the latest library stack and cc8.0.
For now, you could either build from source or let the JIT compiler run in the first CUDA call.