No the inputs are usually a tensor of ints of size batch_size x sequence_length (with integers between 0 and the vocabulary size of the model -1). You can find all expected shapes and output shapes in the documentation, for instance here 104 for BERT.