How to make transformers examples use GPU?
# ❓ Questions & Help

I'm training the run_lm_finetuning.py with wiki-raw dataset. The training seems to work fine, but it is not using my GPU. Is there any flag which I should set to enable GPU usage?

## Details
I'm training the run_lm_finetuning.py with wiki-raw dataset. The training seems to work fine, but it is not using my GPU. Is there any flag which I should set to enable GPU usage?


GPU should be used by default and can be disabled with the `no_cuda` flag. If your GPU is not being used, that means that PyTorch can't access your CUDA installation. 

What is the output of running this in your Python interpreter?

```python
import torch
torch.cuda.is_available()
```<|||||>Thanks for the response. The output is True. Looks like it is using the GPU. But the utilization never crosses 10%. <|||||>And how is your CPU usage? Which GPU are you using? Which settings are you using? (Batch size, seq len...)<|||||>CPU Usage also is less than 10%. I'm using a Ryzen 3700X with Nvidia 2080 ti. I did not change any default settings of the batch size (4) and sequence length. <|||||>@abhijith-athreya  What was the issue? I am facing the same issue. I am encoding the sentences using bert model but it's quite slow and not using GPU too.
<|||||>You need to post some sample code @monk1337, also https://discuss.huggingface.co will be more suited<|||||>@julien-c 

It's working now.

from transformers import BertTokenizer, BertModel, BertForMaskedLM
def assign_GPU(Tokenizer_output):
    
    tokens_tensor = Tokenizer_output['input_ids'].to('cuda:0')
    token_type_ids = Tokenizer_output['token_type_ids'].to('cuda:0')
    attention_mask = Tokenizer_output['attention_mask'].to('cuda:0')
    
    output = {'input_ids' : tokens_tensor, 
              'token_type_ids' : token_type_ids, 
              'attention_mask' : attention_mask}
    
    return output



```
sentence  = 'Hello World!'
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
model     = BertModel.from_pretrained('bert-large-uncased')

inputs    = assign_GPU(tokenizer(sentence, return_tensors="pt"))
model     = model.to('cuda:0')
outputs   = model(**inputs)
outputs
```<|||||>> @julien-c
> 
> It's working now.
> 
> from transformers import BertTokenizer, BertModel, BertForMaskedLM
> def assign_GPU(Tokenizer_output):
> 
> ```
> tokens_tensor = Tokenizer_output['input_ids'].to('cuda:0')
> token_type_ids = Tokenizer_output['token_type_ids'].to('cuda:0')
> attention_mask = Tokenizer_output['attention_mask'].to('cuda:0')
> 
> output = {'input_ids' : tokens_tensor, 
>           'token_type_ids' : token_type_ids, 
>           'attention_mask' : attention_mask}
> 
> return output
> ```
> 
> ```
> sentence  = 'Hello World!'
> tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
> model     = BertModel.from_pretrained('bert-large-uncased')
> 
> inputs    = assign_GPU(tokenizer(sentence, return_tensors="pt"))
> model     = model.to('cuda:0')
> outputs   = model(**inputs)
> outputs
> ```

Hey, I just want to complement here. The current version of transformers does support the call to `to()` for the `BatchEncoding` returned by the tokenizer, making it much more cleaner:

```python
> device = "cuda:0" if torch.cuda.is_available() else "cpu"
> sentence  = 'Hello World!'
> tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
> model     = BertModel.from_pretrained('bert-large-uncased')

> inputs    = tokenizer(sentence, return_tensors="pt").to(device)
> model     = model.to(device)
> outputs   = model(**inputs)
```<|||||>wanted to add that in the new version of transformers, the Pipeline instance can also be run on GPU using as in the following example:
```python
pipeline = pipeline(TASK, 
                    model=MODEL_PATH,
                    device=1,     # to utilize GPU cuda:1
                    device=0,     # to utilize GPU cuda:0
                    device=-1)    # default value which utilize CPU
```<|||||>> wanted to add that in the new version of transformers, the Pipeline instance can also be run on GPU using as in the following example:
> 
> ```python
> pipeline = pipeline(TASK, 
>                     model=MODEL_PATH,
>                     device=1,     # to utilize GPU cuda:1
>                     device=0,     # to utilize GPU cuda:0
>                     device=-1)    # default value which utilize CPU
> ```

And about work with multiple GPUs?