Thanks again! I am now doing
def forward(self, emb):
  #old way   
  tmp = [cnn(emb).squeeze() for cnn in self.cnns]
  seq_representation = torch.cat(tmp, dim=1)

  #new way
  stream_tmp = []
  streams = [(idx, torch.cuda.Stream()) for idx, cnn in enumerate(self.cnns)]
  for idx, s in streams:
    with torch.cuda.stream(s):
      cnn = self.cnns[idx]     #<--- how to ensure idx is in sync with the idx in for loop?
      stream_tmp.append((idx, cnn(emb).squeeze()))
  stream_tmp = [t for idx, t in sorted(stream_tmp)]
  seq_representation_stream = torch.cat(stream_tmp, dim=1)  
  
  #comparing the two
  diff = abs(seq_representation_stream - seq_representation).sum().data[0])
  print(diff)
  assert diff == 0.0
  return seq_representation

In some random batches the assert fails (the diff is very large > 1000 so its not a rounding error).
I am pretty sure it is because the idx in the for loop is not in sync with the idx inside the with torch.cuda.stream(s) block. Sorry this is more of a python question that a pytorch question – but from the documentation, it is not clear how to open multiple streams and concat their results.