Just for future reference if anyone sees this: I solved my problem by using a custom collate_fn for the loader. Instead of using torch.stack I generated a list with all my tiles so the tensor size mismatch was no issue.