I guess you are reshaping the intermediate tensors somewhere in your model and are reducing the batch size to 1.
Your current code is not easily readable, so please feel free to post an executable code snippet to reproduce this error by wrapping it into three backticks ```