Hi,
I have a PyTorch seg training running on a SageMaker-managed EC2, and training errors after 1000 minibatches with the training Dataloader breaking because “RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0”
What could cause this? One bad record in the dataset? Is there a way to build fault-tolerant datasets/data loaders that don’t break if one record is bad?..