The problem is that my input is from a pre-trained model which I want to fine-tune, but at the same time, I want to update parts of the input.
I understand that this is a bit ill-defined/not-nice conceptually. Would be happy to hear any thought you have.