Thanks for the information about the different shapes.
If you cannot create a batch of inputs and concatenate the outputs, the easiest way would be to use a batch size of 1 as you’ve mentioned, until nested tensors are available.



 kontrabas:

I think that I can handle training with samples (bs=1) (with having increased time of training), but will it have any impact later on the inferencing with batches or it should be fine?


A batched inference should not change the behavior of the model, but you would have to face the shape mismatch again I guess. How would the batches and concatenation work during inference, if they are not possible during training?