You could see the timing relatively between the CPU and GPU workload.
E.g. if the GPU workload is small (due to a small model), you might see the overhead of the kernel launches as well as other CPU operations such as list(...), torch.stack(...), data loading, etc.
To isolate the actual runtime of the models in parallel, you could remove the data loading and just profile the models in isolation. Afterwards you could then profile the data loading alone and would have an idea about the speed of the different parts of the code.
Based on your description, my best guess is that the GPU workload is too small for a batch size of 1 and that the mentioned overheads are visible.