Assuming spatialmodel is an instance of  ConvLSTM, you should get an error in unpacking the x.shape in the `forward method using the second code snippet.
Here you are already merging the first two dimensions:
rgb_data = rgb_data.view(b*d, c, h, w)
rgb = spatialModel(rgb_data.to(device))

so that the input to spatialModel would have only 4 dimensions.
However, in the first operation in the forward method you are trying to unpack the shape into 5 values:
b, d, c, h, w = x.shape

which would already raise an error:
x = torch.randn(1, 2, 3, 4, 5)
x = x.view(-1, 3, 4, 5)
a, b, c, d, e = x.shape
> ValueError: not enough values to unpack (expected 5, got 4)

so I guess your posted code snippets might be incomplete. Could you check it, please?