Hi, thanks for your reply
It is a traced model, and it only happens on GPU (on cpu things are even).
I’ve tried this on Pytorch 1.8.1 cuda 10, on both Windows 10 and Ubuntu 20.04 (the former on a Quadro P2000 Mobile with driver 466.11 and the latter has a Titan RTX on 450.102.04) and the results are similarly skewed (the 2nd frame takes 30 seconds on the Titan).
If it is of any assistance in understanding what is going on  these are the profiles from the first 3 frames:

Untitled1770×1200 181 KB

Captured with this block:
with torch.autograd.profiler.profile(use_cuda=True, with_stack=True) as prof:
    inp = [torch.from_numpy(i).float().to(self.device) for i in inputs]
    with torch.no_grad():
        out = self.sess(*inp)
    result = [o.detach().cpu().numpy() for o in out]

Btw is there any way to check that pytorch is loading the correct cuda libraries?
Thank you!