If changing the batch size yields this error, most likely you are using a wrong view or reshape operation.
Could you post the model definition and also try to make sure to keep the batch dimension constant e.g. via x.view(x.size(0), -1) instead of x.view(-1, some_features).