filter through the parameters with some kind of lambda:
params = list(decoder.parameters())
new_params=[]
for p in model.parameters()
#filter here
optimizer = torch.optim.Adam(new_params, lr=args.learning_rate)