How to implement differential learning rates and still ensure "weight_decay" = 0 for the the parameters it should?
I notice that most models are trained with the following parameter groups:
```
no_decay = ["bias", "LayerNorm.weight"]

optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
        "weight_decay": 1e-5,
    },
    {
        "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
        "weight_decay": 0.0,
    },
]
optimizer = AdamW(optimizer_grouped_parameters, lr=5e-3, eps=1e-9)
```

**Questions:**
1. Why do we set `weight_decay` = 0.0 for the `no_decay` named parameters?

2. Assuming we need to set `weight_decay` = 0.0 as such, how would we extend the above to allow for differential learning rates (which may be especially helpful for models that include both an encoder and decoder stack)?

3. Have there been any tests to demonstrate the relative effectiveness of differential learning rates for the various transformer models?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
