Can I use TorchText Iterator output as the input_ids for Hugging Face Transformer?
Hello,

For a given text, `TorchText` iterator, such as the `BPTTIterator`, returns the string of the text after each token is converted to their respective integer ID.

So for example, if the integer ids are assigned as the following manner:
"I" = 53, "like"=753, "dogs" = 2

Then for the string "I like dogs", a `TorchText` iterator would return `[53, 753, 2]`.

Is it okay to use this type of TorchText Iterator output directly as an `input_ids` for Hugging Face Transformers, providing that the Transformer models I use is not the Hugging Face pre-trained ones?

Thank you,

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
