Hey everyone!
I’m currently using gbert from huggingface to do sentence similarity.
The dataset is nearly 3M
The encoding part is taking too long.
for sentence in list(data_dict.values()):
    tokens = {'input_ids': [], 'attention_mask': []}
    new_tokens = tokenizer.encode_plus(sentence, max_length=512,
                                       truncation=True, padding='max_length',
                                       return_tensors='pt',
                                       return_attention_mask=True)
    tokens['input_ids'].append(new_tokens['input_ids'][0])
    tokens['attention_mask'].append(new_tokens['attention_mask'][0])

    # reformat list of tensors into single tensor
    tokens['input_ids'] = torch.stack(tokens['input_ids'])
    tokens['attention_mask'] = torch.stack(tokens['attention_mask'])

    outputs = model(**tokens) # takes too long
    embeddings = outputs[0]

Can someone advise me how to speed up this process? Is it possible to run
outputs = model(**tokens)
on GPU? Would converting the model into an onnx help?
Thank you!