I found anothor solution. The following sample could work, too.
params = list(model.img_encoder.fc.parameters()) + list(model.qu_encoder.parameters()) \
             + list(model.fc1.parameters()) + list(model.fc2.parameters())

optimizer = optim.Adam(params, lr = LEARNING_RATE)