Thank you for your reply, and sorry if unclear.
My question is indeed about parallelization.
I am asking if it is possible to distribute the computations for multiple sentences within a single batch across different cores or nodes - assuming this can speed things up and/or least reduce memory requirement.
Basically, the idea is to split the batch into “sub-batches” that can get distributed, and then re-assemble the outputs from each sub-batch. All this without changing the way the input is passed, I.e. without having to manually split the batches in advance, nor the output shape.
Hope it makes more sense.