Hi, thank you for your reply. I did use model.cuda(idx) and inputs.cuda(idx). Now I create only one model instance, and set  model.cuda(1) and inputs.cuda(1), the exception is still there. I think somewhere it uses the default device which is gpu:0.
Here is some code (creating 10 instances):
class ModelA:
    def __init__(self,gpu=0):
        self.gpu = gpu
        model_path = 'model.pth'
        mixed_precision = True
        model, state = Model.load(filename=model_path)
        self.model = model.cuda(self.gpu)
        self.model = amp.initialize(model, None,
                           opt_level='O2' if mixed_precision else 'O0',
                           keep_batchnorm_fp32=True,
                           verbosity=0)
        self.model.eval()
  def infer(self,inputs):
        inputs = inputs.cuda(self.gpu)
         with torch.no_grad():
            results = self.model(inputs)
         results = results.cpu()
         return results

models = []
for i in range(10):
  models.append(ModelA(i%2)) #10 instances allocated into 2 valid gpus
#receiving requests and running inference with one of these 10 instance.