Hi albanD
I used cuda.synchronize and found that the function that was actually taking the most time was the actual model call, not the loading into gpu calls.
I did
 torch.cuda.synchronize()
 a = time.perf_counter()
 var_mask = var_mask.cuda()
 torch.cuda.synchronize()
 b  = time.perf_counter()
 print(b-a)
 torch.cuda.synchronize()
 a = time.perf_counter()
 var_rgb = var_rgb.cuda()
 torch.cuda.synchronize()
 b = time.perf_counter()
 print(b - a)
 torch.cuda.synchronize()
 a = time.perf_counter()
 output = trainable_model(var_rgb)
 torch.cuda.synchronize()
 b  = time.perf_counter()
 print(b - a)

and got
0.00036394898779690266
0.0005174019897822291
0.8291985339892562
so it looks like just evaluating the model is taking most of the time. is it possible to speed this up? I have 4 GPUs on this computer so running it in parallel should make it faster right?
thanks