If your data and target tensors have a different shape in both Datasets, note that the DataLoader using the ConcatDataset would try to stack these tensors and would thus assume they have the same shape.
E.g. this example works, because the batch size is chosen such that each batch has a valid shape:
class MyDataset1(Dataset):
    def __init__(self):
        pass
    
    def __getitem__(self, index):
        x = torch.randn(10)
        y = torch.randn(4)
        return x, y
    
    def __len__(self):
        return 10


class MyDataset2(Dataset):
    def __init__(self):
        pass
    
    def __getitem__(self, index):
        x = torch.randn(3, 24, 24)
        y = torch.randint(0, 10, (1,))
        return x, y
    
    def __len__(self):
        return 20
    

dataset1 = MyDataset1()
dataset2 = MyDataset2()
dataset = ConcatDataset((dataset1, dataset2))
loader = DataLoader(dataset, batch_size=10)

for data, target in loader:
    print(data.shape)
    print(target.shape)

If you would use shuffle=True, then the DataLoader would yield an error.
You could potentially avoid it by using a custom collate_fn.
However, the first approach might be more suitable (using zip).
How many workers are you using in both DataLoaders? Could you reduce them and check, if this might help the performance?