[Agents] RuntimeError: Invalid device string: 'hakurei/waifu-diffusion'
### System Info

Hey!

I have encountered an issue in the `agents` extra where calling `load_tool()` with `model_repo_id=<sd checkpoint model id>` causes a `RuntimeError` to occur. It would appear that the repo id is being used as a device_id when using the `text-to-image` tool:

```
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
Traceback (most recent call last):
  File "/home/simtoon/transformers/main.py", line 18, in <module>
    img = imggen("cute anime cat")
  File "/home/simtoon/.cache/huggingface/modules/transformers_modules/huggingface-tools/text-to-image/8a3d5357ffa541880148f2425c83ba89f7d56172/text_to_image.py", line 45, in __call__
    self.setup()
  File "/home/simtoon/.cache/huggingface/modules/transformers_modules/huggingface-tools/text-to-image/8a3d5357ffa541880148f2425c83ba89f7d56172/text_to_image.py", line 36, in setup
    self.pipeline.to(self.device)
  File "/home/simtoon/transformers/venv/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 682, in to
    module.to(torch_device, torch_dtype)
  File "/home/simtoon/transformers/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1126, in to
    device, dtype, non_blocking, convert_to_format = torch._C._nn._parse_to(*args, **kwargs)
RuntimeError: Invalid device string: 'hakurei/waifu-diffusion'
```

```
- `transformers` version: 4.30.2
- Platform: Linux-5.4.0-139-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A
```

### Who can help?

cc @Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

- install `transformers`
- install `transformers[agents]`
- call `load_tool` with `model_repo_id` set to a valid sd checkpoint repo id: `imggen = load_tool(task_or_repo_id="huggingface-tools/text-to-image", model_repo_id="hakurei/waifu-diffusion")`
...
`img = imggen("cute anime cat")`
- observe the exception

### Expected behavior

Pull model from hf hub or load from local and use in the txt2img task
Device_string is not looking good.

@LysandreJik might know better than me where this is coming from.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>@LysandreJik afaik this still hasn't been fixed<|||||>Hey @simonSlamka sorry for the late response. I think what you're trying to do is out of scope of what we're trying to do, we haven't designed the remote tools to work by specifying specific checkpoints in this way.

For this tool in particular (but this will need to be adapted to remote tools as these can be community contributed), here's how you would go about it:

```py
from transformers import load_tool

imggen = load_tool(task_or_repo_id="huggingface-tools/text-to-image")
imggen.default_checkpoint = "hakurei/waifu-diffusion"

img = imggen("cute anime cat")
```

I think the best way to leverage a given checkpoint here would be to clone the existing remote tool, replace the checkpoint, and update the generation settings so that they work best with the checkpoint you have in mind. So you would have a remote tool of your own, for example `simonSlamka/anime-text-to-image` that could be used to replace the existing image generation tool in the toolbox (or provided as an additional tool with anime as its focus).

Hope that helps!<|||||>Hi,

Thanks a lot for your assistance and guidance. I will do what you suggested.

Have a nice day!