Many things here have been said, and all are True, however I would like to weigh in on this:


Using the GPU will result in faster results most likely if you can use it


If you use a GPU, try to use a DataLoader and make the Dataset run the tokenization, this will make sure GPU is always busy


Be careful about batching on real data, if there’s too much padding involved it might actually decrease performance. (If you have no idea what’s in your data, I recommend batch_size=1 do avoid memory overflow + slower perf overall)


ONNX will improve speed on CPU, and can become very competitive to GPU but it’s unlikely to be more performant. (But it’s much easier to scale by adding more CPU for compute)


Make sure you are using Fast tokenizer and not the slow ones


Measure everything you do, and try to use not too simple use cases as it might not represent accurately what it going to happen on real data. (Use like the first 100 examples of real data over and over to get a better sense)


You can use pipeline to get probably a much easier to use API which will take care of most of this if you use a Dataset.


from transformers import pipeline                                              
from torch.utils.data import Dataset                                           
import tqdm                                                                    
                                                                               
data_dict = {i: "This is a test" if i % 2 else ("This is a longer test" * 20) for i in range(1000)}
                                                                               
                                                                               
model_name = "nlptown/bert-base-multilingual-uncased-sentiment"                
                                                                               
pipe = pipeline("text-classification", model=model_name, device=0)             
                                                                               
                                                                               
class MyDataset(Dataset):                                                      
    def __init__(self, dic):                                                   
        self.dic = dic                                                         
                                                                               
    def __len__(self):                                                         
        return len(self.dic)                                                   
                                                                               
    def __getitem__(self, i):                                                  
        sentence = self.dic[i]                                                 
        return sentence                                                        
                                                                               
                                                                               
dataset = MyDataset(data_dict)                                                 
                                                                               
                                                                               
for outputs in tqdm.tqdm(pipe(dataset)):                                       
    pass       

Here is the full low-level example you can use to tweak parameters if batching makes sense in your use case:
from transformers import AutoTokenizer, AutoModel                                 
import torch                                                                      
from torch.utils.data import Dataset, DataLoader                                  
import tqdm                                                                       
                                                                                  
data_dict = {i: "This is a test" if i % 2 else ("This is a longer test" * 20) for i in range(1000)}   
                                                                                  
                                                                                  
model_name = "nlptown/bert-base-multilingual-uncased-sentiment"                   
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)              
model = AutoModel.from_pretrained(model_name)                                     
device = torch.device("cuda:0")                                                   
model = model.to(device)                                                          
                                                                                  
                                                                                  
class MyDataset(Dataset):                                                         
    def __init__(self, tokenizer, dic):                                           
        self.tokenizer = tokenizer                                                
        self.dic = dic                                                            
                                                                                  
    def __len__(self):                                                            
        return len(self.dic)                                                      
                                                                                  
    def __getitem__(self, i):                                                     
        sentence = self.dic[i]                                                    
        tokens = tokenizer(sentence, return_tensors="pt")                         
        return tokens                                                             
                                                                                  
                                                                                  
dataset = MyDataset(tokenizer, data_dict)                                         
                                                                                  
                                                                                  
def collate_fn(batch):                                                            
    return tokenizer.pad([{k: v.squeeze(0) for k, v in item.items()} for item in batch])   
                                                                                  
                                                                                  
dataloader = DataLoader(dataset, batch_size=1, collate_fn=collate_fn, num_workers=4)                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                          
for tokens in tqdm.tqdm(dataloader):                                                                                                                                                                                                                                                      
    tokens = {k: v.to(device) for k, v in tokens.items()}                                                                                                                                                                                                                                 
    outputs = model(**tokens)  # takes too long                                                                                                                                                                                                                                           
    embeddings = outputs[0]