Before finetuning, I’d echo @Williamsdoug’s suggestion and try an existing model. The training data really is a vital factor.
If we check out the model used by default for sentiment analysis:
c1 = pipeline(task="sentiment-analysis")
c1.modelcard  # None :-(
c1.model  # DistilBertForSequenceClassification

We can search on Huggingface’s hub for a model that matches “DistilBertForSequenceClassification”

image1118×288 39.9 KB

I only see one result, and there’s no more obvious info there. But a good guess, by default, is that a model was trained on scraped web text, which might not have great performance for the domain you’re looking for.
If we test out this DistillBert model a bit more:
for i in range(2, 25):
    s = f"this is so g{'o'*i}d!"
    print(s, c1(s))

outputs:
this is so good! [{'label': 'POSITIVE', 'score': 0.9998476505279541}]
this is so goood! [{'label': 'NEGATIVE', 'score': 0.9995097517967224}]
this is so gooood! [{'label': 'NEGATIVE', 'score': 0.9922532439231873}]
this is so goooood! [{'label': 'NEGATIVE', 'score': 0.9786434173583984}]
this is so gooooood! [{'label': 'NEGATIVE', 'score': 0.9764403700828552}]
this is so goooooood! [{'label': 'NEGATIVE', 'score': 0.9486815333366394}]
this is so gooooooood! [{'label': 'NEGATIVE', 'score': 0.9464263319969177}]
this is so goooooooood! [{'label': 'NEGATIVE', 'score': 0.9400624632835388}]
this is so gooooooooood! [{'label': 'NEGATIVE', 'score': 0.9257988929748535}]
this is so goooooooooood! [{'label': 'NEGATIVE', 'score': 0.9494100213050842}]
this is so gooooooooooood! [{'label': 'NEGATIVE', 'score': 0.931290328502655}]
this is so goooooooooooood! [{'label': 'NEGATIVE', 'score': 0.947573721408844}]
this is so gooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.9173546433448792}]
this is so goooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.935723602771759}]
this is so gooooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.8896605968475342}]
this is so goooooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.9372522234916687}]
this is so gooooooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.8814820647239685}]
this is so goooooooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.9387907385826111}]
this is so gooooooooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.8939523696899414}]
this is so goooooooooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.9408086538314819}]
this is so gooooooooooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.9059429168701172}]
this is so goooooooooooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.9447947144508362}]
this is so gooooooooooooooooooooooood! [{'label': 'NEGATIVE', 'score': 0.9063981175422668}]

Bummer.
But try loading up a different one, like a sentiment model trained on Twitter (again, as @Williamsdoug suggests!)
c2 = pipeline(
    task="sentiment-analysis", model="cardiffnlp/twitter-roberta-base-sentiment"
)

The outputs for this model aren’t as clear:
c2("hello")  # [{'label': 'LABEL_1', 'score': 0.6034973859786987}]

… so we can follow the example on the model’s page 3 and download a label mapping:
# get label mapping for c2
labels = []
mapping_link = f"https://raw.githubusercontent.com/cardiffnlp/tweeteval/main/datasets/sentiment/mapping.txt"
with urllib.request.urlopen(mapping_link) as f:
    html = f.read().decode("utf-8").split("\n")
    csvreader = csv.reader(html, delimiter="\t")
labels = [row[1] for row in csvreader if len(row) > 1]
# labels = ['negative', 'neutral', 'positive']

Then, we can try the same test:
for i in range(2, 25):
    s = f"this is so g{'o'*i}d!"
    res = c2(s)
    # map LABEL_X -> pos./neut./neg.
    res[0]["label"] = labels[int(res[0]["label"][-1])]  
    print(s, res)

outputs:
this is so good! [{'label': 'positive', 'score': 0.9917314648628235}]
this is so goood! [{'label': 'positive', 'score': 0.9901121854782104}]
this is so gooood! [{'label': 'positive', 'score': 0.989584743976593}]
this is so goooood! [{'label': 'positive', 'score': 0.9901219606399536}]
this is so gooooood! [{'label': 'positive', 'score': 0.9901283383369446}]
this is so goooooood! [{'label': 'positive', 'score': 0.9899706840515137}]
this is so gooooooood! [{'label': 'positive', 'score': 0.9893210530281067}]
this is so goooooooood! [{'label': 'positive', 'score': 0.9891370534896851}]
this is so gooooooooood! [{'label': 'positive', 'score': 0.9892650246620178}]
this is so goooooooooood! [{'label': 'positive', 'score': 0.9897728562355042}]
this is so gooooooooooood! [{'label': 'positive', 'score': 0.9892972111701965}]
this is so goooooooooooood! [{'label': 'positive', 'score': 0.9887627959251404}]
this is so gooooooooooooood! [{'label': 'positive', 'score': 0.9891249537467957}]
this is so goooooooooooooood! [{'label': 'positive', 'score': 0.9889877438545227}]
this is so gooooooooooooooood! [{'label': 'positive', 'score': 0.9879323840141296}]
this is so goooooooooooooooood! [{'label': 'positive', 'score': 0.987424910068512}]
this is so gooooooooooooooooood! [{'label': 'positive', 'score': 0.9873157143592834}]
this is so goooooooooooooooooood! [{'label': 'positive', 'score': 0.9900829195976257}]
this is so gooooooooooooooooooood! [{'label': 'positive', 'score': 0.9894198179244995}]
this is so goooooooooooooooooooood! [{'label': 'positive', 'score': 0.9894325137138367}]
this is so gooooooooooooooooooooood! [{'label': 'positive', 'score': 0.9897305369377136}]
this is so goooooooooooooooooooooood! [{'label': 'positive', 'score': 0.9894993901252747}]
this is so gooooooooooooooooooooooood! [{'label': 'positive', 'score': 0.9883342385292053}]