A few random thoughts…


If the module order and layer sizes are fixed, then the only uncertainty is the input size, since if we know how many units there are in layer1, then we know what the input size of layer2 should be. Keras only requires you to specify the input size of the first layer. The rest are inferred at model creation time. That said there will be cases where you might want to have a layer whose size is a function of its input size, for example, a layer with input_size*10 units.


nn.LSTM can initialize its own hidden state at runtime, but you can’t specify a custom initialiser function. Maybe we could do something like that for the weight arrays.


Stochastic depth could be implemented with a random skip class that would consume a class from the ModuleList at each run, but choose at random whether to pass the input data straight through or whether to pass it through the module. A form of layer dropout if you like.