@BramVanroy I think you can use a single optimizer, but specify a different learning rate for the parameters you want, like so:
optimizer = optim.Adam([
                {'params': model.parameters()},
                {'params': model.embeddings.pos_tag_embeddings(), 'lr': 5e-5}
            ], lr=2e-5)