Have you tried to detach the position of tokens you don’t want to have gradient?
The detach() operation makes them not accumulate the gradients in the backward path.
Just detach the position you want to block the gradient after the embedding layer.
This may be easier done than customizing autograd function.