Consider the following snippet:
from time import time
import torch

for run in range(5):
    start = time()
    torch.empty(1, device="cuda")
    end = time()
    print(f"{run}: {end - start:.2g} seconds")

On my machine this results in the following output:
0: 1.6 seconds
1: 3e-05 seconds
2: 7.9e-06 seconds
3: 7.4e-06 seconds
4: 7.2e-06 seconds

It is pretty clear that the first time a tensor a moved to the GPU some startup or initialization is happening. Is it possible to manually do this upfront?

I’m asking this since I have multiple tests in my test suite that require CUDA. Depending on which test is run first, it indirectly performs the startup is therefore slow. If have a flag to skip slow tests, but the test in itself is not slow.