I would just create a look using your Dataset without wrapping it into a DataLoader:
dataset = ...
for idx, (data, image) in enumerate(dataset):
    print(idx)

This loop should raise an exception once the KeyError is raised. Using the last printed idx you could narrow down, which samples creates this issue.