Hi. I am training an RNN encoder-decoder to make predictions of the trajectory of agents. I am using the KITTI dataset. The training dataset consists of 21 image folders and each folder has a variable number of images. So, this means that the batch size changes during training, for example:
A batch size of one, when training an encoder, means one sequence. Keeping that in mind, let’s say I take as input 5 images and, while decoding, also works on 5 images. So, that is a total of 10 images. So, a single sequence consists of 10 images, hence, a batch size of one is the same as 10 images in this case. Now, I have a folder of 30 images, so I can work on 3 sequences, hence my batch size is 3 for this folder. However, the next folder has 50 images, so that means the batch size becomes 5.
So, I guess my question is whether this is okay or not? Is there any harm while training like this?