Your GPU might go into an idle state and the wake up might take some time. You could enable the persistence demon as described in the docs 2:

This approach would prevent the kernel module from fully unloading software and hardware state when no user software was using the GPU.