Thanks for the detailed explanation. I agree that at some point you just have to go back to framework-specific implementations, but I was curious how the distinction will be made for the rest of the library as well as for datasets. In case of the latter, I think that datasets also casts to numpy arrays as intermediate states and optionally returns tensors in the format that is requested by the user. (This has some underlying issues 1, too, because there is no one-on-one dtype conversion between numpy and torch.)
That begs the question: from a user-perspective, will this get too complicated when putting all the parts together? Will it be clear for users which part of the library are framework specific (pipeline, models) and which ones aren’t (datasets, tokenizers, trainer (?)).