I want to set up a network that may have lots of complicated skip connections, but will still be feed forward (no recurrent connections). Each node will still be a “classic” node in the sense that it will just have a weight for each connection, and then multiply each weight by the corresponding input, sum them, and put it through a nonlinear function.
For example, it might look like this:

hbeh9.jpg525×540

However, I don’t think I can easily use something like the  nn.Linear  module to calculate outputs, because (in the picture for example), the red nodes get inputs from the green nodes  and  the pink nodes, but the green nodes also need the pink nodes’ outputs to calculate.
I can think of some ways to do this, but they’re real ugly and pytorch usually has a smart way of doing things. Is there a good way to do this? thanks!