I am using GeForce GTX 1080 Ti. I test with small models( about 40000 parameters). I think, my GPU can enough compute. I also use the @profiler to measure time. The result :
Timer unit: 1e-06 s
Line #      Hits    Time         Per Hit   % Time          Line Contents
   204      4689    5770628.0   1230.7     13.0            output_2 = net_1(data);
   205      4689    2219502.0    473.3      5.0            output_1 = net(data);
   206      4689    2166653.0    462.1      4.9            output_3 = net_2(data);
   207      4689    2156971.0    460.0      4.9            output_4 = net_3(data);

Running with 1 command line:
Line #      Hits    Time         Per Hit   % Time          Line Contents 
204         4689    5422347.0   1156.4     18.9          output_2 = net_1(data);


i think the total time is not the same.
This means: this doesn’t run in parallel