Solved by ptrblck in post #2 
              
              
                Assuming the error is raised in one of the prompt_encoders, check of their internal modules are already pushed to the device and if not either call model.to('cuda') on the entire model or on each of the prompt_encoders.