@andreasceid Thank you very much! The encoder are in a ModuleList. I put more of my code in the question including how they are called in the forward of the container Module. The container module actually wrap a transformer model (T5) which is freezed and the result of forward pass on encoders are fed into it. I am someway beginner with Pytorch and Transformer. For example some parts of my LSTM enoders such as having both input_id parameters and net_ids might be redundant?
Suppose input ids for whole model are [1, 2, 3, 4, 5, 1, 2, 3, 4, 5, ....] in batch. the ids of encoder 1 are [3, 4] and the ids for encoder 2 are [1,3, 5]. where 3 is common between them. Each encoder must update embedding for its corresponding input. However, 3 exists in both of them. Maybe I should merge the results for 3 in the loop in forward, I don’t know. I thought maybe they must refer to a shared embedding space. Please guide me for this problem.