It looks like your train_data variable is used for two different things. Are you sure you passed the instance of your CLIPDataset to the Trainer? Cause it looks like the elements of the training dataset of the Trainer are not dictionaries from the error message.