I like this. Especially the focus on customizability on the one hand and having a training pipeline on the other.
Even though I agree with the direction you want to take considering separating the frameworks, I wonder: a few months back it seemed that there was a lot of effort going into unifying everything and being framework-agnostic. What changed? Does that make things too complex? Does it limit your options too much?
One thing that I am not sure about is the integration of the tokenizer in the pipeline, though. Usually we would either preprocess the data (e.g. using datasets) or tokenize and pad in a collate_fn in the dataloader, which can be parallellized. As your example is currently written, it seems that you just get the raw text input from the dataloader (i.e. batch of sentences = list of strings) and pass it to nlp.forward, which is then responsible for tokenization and forwarding through the model at every step. I think this would make everything slow and inefficient, and also un-PyTorch-y if that’s a word. The preprocessing of the data should happen a lot earlier imo.
Just my two cents!