Thanks for you reply. I did try it by reshaping it to (1, 3, 128, 128) and that worked. However I was stacking up LSTM cells instead of a layered LSTM.
Yes using CNN for feature extraction will work but I want to avoid it for a specific reason. I deliberately want to pass an image to the LSTM.