I finally understood, after going-over some projects implementing something that I am trying to do too and with the docs help:
There is the 4th input tensor to be supplied:

target_lengths: Tuple or tensor of size (N).
Lengths of the targets

Thus, I can pad at the end with whatever I want while specifying the original size of targets/labels before padding.