The error was in the following line of code:
 labels_array = np.array(self.dataframe.loc[self.dataframe['image_id'] == img_id, ['label']])
        labels = torch.tensor(labels_array, dtype=torch.int64)
        target['labels'] = labels

This resulted in an embedded list of targets. Changing it to
        final_labels = []
        labels_array = np.array(self.dataframe.loc[self.dataframe['image_id'] == img_id, ['label']])
        for value in labels_array:
            final_labels.append(value[0])

        labels = torch.tensor(final_labels, dtype=torch.int64)
        target['labels'] = labels

solved it for me. The solution above could probably be done more elegantly but this workaround worked for me.