oh right, you should run them on different cuda streams https://pytorch.org/docs/master/cuda.html#streams-and-events 23.
but in general you should expect slow down in paralleling tasks. there will always be overhead and usually not all tasks can run with 100% speed.