I see! You are extracting the weights after building the module, and then feeding into it. Now that makes sense! Ignore my previous comments about them not being trained. They are indeed being trained.
However, I don’t see how you get two sets of weights in original code. base_weights = OrderedDict((name, param) for (name, param) in model.named_parameters()) would just give you the same set of weights .
What I would do is still the snippet I wrote above. Just that when you manually train, the set of parameters should include both set l1 and l2's parameters (or other names/types/networks whatsoever).