From section 3.2

we train the model to output 2H-D vector, representing predicted x- and y-positions for each of H trajectory point

it seems that the model output is [H, 2H/3H] where H is the no. of trajectory points in the time horizon. So I’d expect the LSTM to receive [H, B, 128] input tensor.