That strategy would probably work. If you can afford to do two embeddings, you can use .clamp and then torch.where to get the right embeddings.
I’d venture that sorting the entries (to have the fixed embeddings in one range and the trainable in a separate) is likely more efficient than having a mask, but I think the mask idea alluded to above might also be a good choice.
Best regards
Thomas