How to speedup BERT eval
## ❓ Questions & Help
Is there a simple way to speedup `.eval()` when using the BERT model
Specifically I am using `BertForSequenceClassification`. I have finetuned a the model separately on my own data and I am trying to get hidden representations after doing `model.eval()` as follows:
`last_hidden_layer, all_hidden_states = model(input_ids)`

However for each input it is taking about 2.3 seconds on `cpu` and 2.6 seconds on `gpu`. Is there a way where I can do faster than this?
Did you try using DistilBERT? Inference should be ~ 60% faster<|||||>Turns out I wasn't using the using the gpu correctly. I moved the model and the inputs to gpu by doing `.to(device)` and it became 100x faster. Thanks for the suggestion.