While it’s certainly an interesting proposal, my concern is that it’s pretty much impossible to identify which modules does someone want to use. I’d imagine that the rule for re-using auto-created modules would be something like: keep a dict of buckets where keys are module types and values are lists of instantiated modules. Upon ith use of some kind of a module (let’s say Linear), go to the Linear instance list and pick the ith module. Use this to compute the function. There might be a better way that I didn’t think of, that’s just what I assume in the remaining part of this post. Now, consider these two cases:

“Stochastic depth”

def forward(self, x):
    if bool(random.randrange(2)):
        y = ann.Linear(ann.Linear(x, 200), 200)
    else:
        y = ann.Linear(x, 200)
    return ann.Linear(y, 200)

In this case taking the first branch will use Linear instances 0 and 1 to compute y, and instance 2 to compute the result. In case the second branch is taken, instance 0 will be used to compute y and the result will be computed using instance 1. I can imagine myself writing such code, and that certainly wouldn’t be my intention.

Reusing a module

Since modules are identified by their use, it’s impossible to use the same module twice (this is quite important in many models). This could be alleviated by naming the modules but I feel that this makes the API unnatural compared to just using self.my_module(x).

Thanks for the proposal and let me know what you think!