title: How to run multiple pytorch layers in parallel, when they are of the same graph depth?
body: <p>Implementing <a href="https://arxiv.org/abs/1910.13051" rel="nofollow noreferrer">this paper (ROCKET)</a>, I ended up with <a href="https://github.com/noamzilo/timeseries_classification_random_conv/blob/master/EcgModel.py" rel="nofollow noreferrer">this code</a>, which works.</p>
  <p>The main issue I have with it is in summary <strong>How to run multiple pytorch layers in parallel, when they are of the same graph depth?</strong></p>
  <p>In the paper, 10000 convolutions with different (randomized) parameters are used on the same input to create 10000 corresponding feature maps, on which simple feature extractors are used.</p>
  <p>Implementing this simplifies to</p>
  <pre><code>class MyModel(pl.LightningModule):
      def __init__(self):
          ...
          num_kernels = 10000
          for _ in range(num_kernels):       
              self.convs[_] = nn.Conv1d(...)
  
  
      def forward(...):
            for i, layer in enumerate(self.convs):
                res = layer(data)
  </code></pre>
  <p>This just runs a Pythonic 10000-for-loop, each iteration runs its corresponding convolution.</p>
  <p>How do I utilize Pytorch to run those 10000 convolutions efficiently, without waiting for each one to end before dispatching the next one?</p>
  <p>This also generalizes for any non-fully-sequential Pytorch models.</p>