This is strange as usually fully connected layers do not depend on the batch size during training, as the data usually has a shape like (N, D) where N is the batch dimension and D is the hidden dimension or sequence length.
What is the purpose of the x.view(-1)? Is the data to be interpreted as a time series? Otherwise mixing data across the batch dimension is strange.
You might want want to consider whether the batch dimension used here actually corresponds to the meaning of a batch with the dataloader, as typically samples within a batch are not mixed (with the exception of layers like batchnorm). If you decide that the use of batch is actually correct here, you can try some common techniques like simply padding the input for edge cases where the data does not fill the expected input size.