My particular use-case originates from reinforcement learning with pixel observation. Lets assume that I sample 16 agent-environment-interactions and use a sequence length of 8.
I’d feed a tensor of shape (16, 3, 84, 84) to a stack of convolutional layers. After that the data needs to be reshaped into sequences of length 8. Doing so by using sequence first will mess up the data. (reshape(8, 2, -1)). Using the batch size as first dimension works well concerning the reshape.