Another option 60 may be to use (abuse?) a Conv1d  layer, using the groups argument to handle the parallel network simultaneously, and using a kernel size of 1 to get it to act as a a Linear layer.
Would that be equivalent? Should I expect a speedup from this approach?