I am not familiar with this work, but your code reflects what you want to do.
Note that if you don’t want these pre-trained models to be updated.
You could “hide” them to the optimizer by storing them in an actual list self.experts = [model1, model2, model3] or set requires_grad to False for all their parameters:
for m in [model1, model2, model3]:
  for p in m.parameters():
    p.requires_grad=False