Two ways:


This could probably be done by introducing an additional masking embedding with n_dims=1 and have
the embedding of a particular word ‘1.0’ if it should use aux embeddings and ‘0.0’ if not.
Set the require_grads to False and do en element wise multiplication between the
aux embedding and the mask embedding.


Calculate a mask (Boolen tensor) that determine if one should use the aux embedding for each particular word
and use it to mask the input before embedding.


Hope this helps