I did not get this issue. I think we were probably making the problem complex so I’d try to rephrase it. The fundamental question here is, when making a customized dataset, what is the best way to return the sequence (with variable lengths) and its length?
For example, it would be something like
input: ["some string", "some other string" ...., "final string"], with a max length max_len
embedding mapping: {"some": 0, "string": 1, ...}
task: return embedded sequences, with their original lengths if doing padding.
Hopefully this is clearer.