Thanks for the clarification. Since my model uses only a small subset of the sub-modules dependent on the input, it could be more efficient if only the used parameters are updated.
However, since there are only parameter updates (no forward/backward passes) for the unused modules, it might not be the bottleneck of the entire process as you mentioned. (If it is a bottlneck, then I’ll use separate modules with their own parameter updates.)
Thanks!