Thanks for getting back to me Sylvain.
My source data looks a bit like this:-





id
created_at
text
sentiment




0
77522
2020-04-15 01:03:46+00:00
RT @RobertBeadles: Yoð¥\nEnter to WIN 1,000 …
positive


1
661634
2020-06-25 06:20:06+00:00

#SriLanka surcharge on fuel removed!\nâ½ð…
negative


2
413231
2020-06-04 15:41:45+00:00
Net issuance increases to fund fiscal programs…
positive


3
760262
2020-07-03 19:39:35+00:00
RT @bentboolean: How much of Amazon’s traffic …
positive


4
830153
2020-07-09 14:39:14+00:00
$AMD Ryzen 4000 desktop CPUs looking âgreatâ…
positive



From this I extract the training data:-





id
created_at
text




10
223041
2020-04-27 00:41:06+00:00
RT @PipsToDollars: Earnings $AMZN $TSLA $MSFT …


147
808963
2020-07-08 19:41:22+00:00
Baidu $BIDU Has A Weak #Technical Analysis Sco…




I turned the sentiment values into numbers:-
y_train.head()
10     2
147    1
342    1
999    3
811    3
Name: sentiment, dtype: int64

I encoded the data using the following code:-
checkpoint = "distilbert-base-cased"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)

sequences = X_train.loc[:,"text"].tolist()

encoded_train = tokenizer(sequences, padding="max_length", truncation=True, return_tensors="tf")

I had a look at some of the data types:-
encoded_train is of the following type:  <class 'transformers.tokenization_utils_base.BatchEncoding'>
np.array(y_train) is of the following type:  <class 'numpy.ndarray'>
y_train is of the following type:  <class 'pandas.core.series.Series'>

Hopefully, some of the above make the problem a little clearer