If an architecture has local branching in the convolution, the most famous example being the Inception blocks where multiple convolutions at different scales are being taken and the results combined, then in PyTorch each convolution will be sequentially evaluated. For example, in the example implementation of an inception block 4. Doesn’t that mean PyTorch will be much slower than Tensorflow for this kind of architecture?
Or, does it not matter because the convolutional are expensive enough they can’t be evaluated in parallel anyway? ie in practice we don’t actually care?