Even though the python interface is sequential, you should know that the cuda api is completely asynchronous.
This means that if you run 3 convolutions in pytorch (without explicit synchronization), the 3 will run at the same time (or one after the other depending on your GPU compute capacity).
You can see this for example when trying to time cuda operations, if you don’t put any explicit synchronization, the call to the cuda function will return almost imediatly. The one that is going to be slow is the one that will make use of the results of the computation as this one will have to wait for the computations to be finished.