I have a LSTM model trained for a batch size = 512. This means that 512 hidden states are initialized for each sample in the batch. Now during prediction if I give a batch_size=100 it throws an inconsistent size error probably because it cannot initialize the hidden states for the 100 samples since we have already set it in init_hidden. It does not throw an error if I pass each sample in the batch as one sample to the model during prediction. The output in such a case is broadcasted to the batch_size. So my question is whether my understanding is correct? Cant I do a batch prediction for a model?
Thanks,
Kaushal