Yes, I agree with you. This type of setup largely doesn’t make sense.
Actually, I wanted to use some autoregressive model such as XLNet, but for my specific data I don’t have XLNet model. So I don’t have an appropriate tokenizer. I was having this wild thought, if I can use some pre-existing tokenizer(which trained on similar data to that of mine) and input it to XLNet model. Argument being(as you mentioned) - anyhow layers would learn some thing, in the worst case they will learn from scratch similar to finetune setting.
But yeah, I also think this can’t/shouldn’t be done as we wouldn’t know how worse the scenario has become.