Hi all,
I’m trying to do some ablation studies (or sensitivity studies in some other jargon), turning on and off different components of my model.
For now I’ve assumed that to avoid passing the input through a certain layer (say an nn.Embedding) it would be sufficient to avoid gradient contributions from that component of the model.
That will evidently make my model parameters larger than necessarily, with some of the layers not being updated past their initialisation.
Something tells me this is not exactly the best way to do these things. What would be your take? Would you rather define a specific model just for that component you want to turn on/off?