Then I would argue that it’s not an issue of padding. Padding is used to “fill up” sequences to enable efficient batch processing. Ideally, a network learns that padding (e.g., index 0) means nothing and can/should be ignored.
For you, there is a semantic meaning that a sequences hasn’t started yet. So your batch would look like:
[4, 4, 1, 2, 3]
[4, 1, 2, 3, 0]
[1, 0, 0, 0, 0]

Where index 4 represents “nothing happened yet” or something.
I’m not really sure if this makes semantically sense, but I would still argue that it’s not the same as padding.