Hello there!
I was able to fine-tune my own model for text classification (from a distilbert-base-uncased-finetuned-sst-2-english · Hugging Face 1 model). Everything works correctly on my PC.
Now comes the app development time but inference - even on a single sentence - is quite slow. I am processing one sentence at a time and using the simple function predict_single_sentence(['this is my input sentence'])

tokenizer = AutoTokenizer.from_pretrained('C:\\Users\\mytrainedmodel')
mymodel = TFAutoModelForSequenceClassification.from_pretrained('C:\\Users\\mytrainedmodel')


def predict_single_sentence(text_list, model, tokenizer):  
    #tokenize the text
    encodings = tokenizer(text_list, 
                          max_length=280, 
                          truncation=True, 
                          padding=True)
    #transform to tf.Dataset
    dataset = tf.data.Dataset.from_tensor_slices((dict(encodings)))
    #predict
    preds = model.predict(dataset.batch(1)).logits  
    
    #transform to array with probabilities
    res = tf.nn.softmax(preds, axis=1).numpy()      
    
    return res

Despite my big RTX, inference is quite slow (about 5 sec for a single sentence). Am I missing something here? Should I create my own pipeline to speed things up?
Thanks!