I should’ve mentioned that you can create the transform as transforms.Resize((224, 224)). If you pass a tuple all images will have the same height and width.
This issue comes from the dataloader rather than the network itself. When the dataloader creates the batches it expects all tensors to have the same shape.