Okay, so model loading is a red herring. It’s always the first copy that is very expensive regardless of the size.
import time
import torch

t0 = time.perf_counter()
torch.tensor((10), device="cuda")
time.perf_counter() - t0
> 5.2702017779956805
t0 = time.perf_counter()
torch.tensor((10), device="cuda")
time.perf_counter() - t0
> 0.00011218398867640644

Is this initial cost unavoidable?