Hi everyone,
I’m wanting to summarize pretty long texts (300 to 5000 words), I have about 30k examples to work with, and I have few questions before I get started, to avoid heading off in the wrong direction.
Now I understand there are models like BART and T5, that are specifically tailored to such tasks. However there is also the option of using seperate models (say BERT and GPT2) and stitch those together via EncoderDecoderModel().
Now, to me it looks like either of those approaches should work fine. Which one will work better can probably only be found out by trying.
Is this assumption correct ?
Another thing I’m wondering about is which of these is going be more costly to train in terms of computing ressources. I’d assume that a single model approach (i.e. T5 oder BART) would use much less memory but maybe I’m missing something?
Is there theoretically a way to train BERT and GPT2 seperately in the EncoderDecoderModel() approach?
I know one can fine-tune this models on their own but is it still gonna be feasible to use them as EncoderDecoder in the model and expect good summarier after that? (I don’t think so but thought I’d ask).
Any clarifications on thos points would be greatly appreciated 
Personally, I’m partial to the EncoderDecoderModel() approach, as there are BERT and GPT-2 models available in my target language, where there is only a small T5 model (not BART) available in the same language.