@apaszke, I was not very clear in explaining how it might work. The idea is to create a network in exactly the same way as we do now, but do it in the “forward” call instead of the “init” call. When we create the network in the forward call we remember the exact and static order of computation and modules used in the ModuleList.  The order of module invocation does not change during the training and all items in the ModuleList remain the same. You can reuse any sub-modules and optionally share the weights (this could be specified as an additional parameter) to point to the same or a different instance in the ModuleList from an ann module. So, each instance of ann module is basically an integer number pointer to an instance in the static ModuleList that does not change during the training. This integer number is used internally just like the name of modules that are declared in the “init” call now.  If the order of computation does not change, just like the JIT graph does not change during the training, the ModuleList would let you reference the sub-modules line by line in the forward call and allow an easy debugging during the training.
It is just a simple idea, I think there is a room for its improvement that would probably give even more expressive power and possibly something more dynamic without a need to pre-declare the structures.