@pbelevich You are right. But I am still wondering why the error occurs on the second use?
The above error occurs when the second usage(namely model(*args)) happens. The first usage is getting the remote params when creating DistributedOptimizer.
Also even I remove the first usage(code as follows), the same error occurs when the model begins to deal with the second batch(The first batch is ok). So I thought the rref object can only be used once.
# remove the rref object from optimizer
opt = DistributedOptimizer(
        optim.SGD,

        # Just get the local params and wrap it with RRef,
        # because I need no update to the remote bert params
        # and bert.requires_grad = False
        [RRef(prompt_embedding_policy.params)],  

        # model.parameter_rrefs(),  # get the remote params, will use self.language_model_service_rref)
        lr=0.05,
    )
...
# Now the same error occurs when the second batch begins to be dealt with.
for batch in data_loader:
    logits = model(batch)  # note just an example here