But the dimensions I am using in the lstm are the sequence length I specified and input_dim shoudl correspond to the dimension of the images, no?
I tried adjusting the sizes for the lstm manually as you stated and it works for the training set, but e.g. for the test set, the output size of them CNN is [4, 1, 28, 28] and then it fails again.