Hi,
I am trying to fine tune T5 for conditional text generation, and it works perfectly. But, I would like to have more control over the layers I am training i.e. freeze and unfreeze certain layers. For example, I am also using GPT2 for the same use case and I do something like this:
# - Freeze selective layers:
# - Freeze all layers except last n:
for parameter in model.parameters():
    parameter.requires_grad = False

for i, m in enumerate(model.transformer.h):        
    #Only un-freeze the last n transformer blocks
    if i+1 > 12 - config['UNFREEZE_LAST_N']:
        for parameter in m.parameters():
            parameter.requires_grad = True 

for parameter in model.transformer.ln_f.parameters():        
    parameter.requires_grad = True

for parameter in model.lm_head.parameters():        
    parameter.requires_grad = True

Now, I am able to freeze all the layers in T5, but I have trouble unfreezing the last n layers, since T5ForConditionalGeneration has no attribute “transformer”. Does anyone has an idea on how to achieve this, because I couldn’t find anything about this issue?