It seems I need to put all encoders to device.
      wrapped_model.to(device=device)
      for encoder in wrapped_model.prompt_encoders:
            encoder.to(device=device)

However, when there was a single encoder or a list of encoders but including one encoder, I didn’t need to explicitly put it on device, but for the list of encoders i must!