If you are using DataParallel the assumption is all input tensors have the same dimension along the first (batch) dimension. Otherwise the splitting behavior becomes tricky to reason about.
What are the input shapes (and the meaning of the dimensions) being passed and is DataParallel being used?