Yes it can be done. But you’ll have to 1) keep a separate weight tensor as parameters, 2) at each training loop build a larger containing weight, fill certain indices with the values from the former tensor with other entries set to 0, and 3) use the functional interface to compute the result.