AlekseyDorkin:

So the question is how do you implement the case where multiple augmented sentences are returned by the function? Does your sentence feature become a sequence of strings? Or do you do something else?


In the documentation, I read that I can use map with batched=True to change the size of a dataset, returning fewer or more instances; unfortunately, I can not find it now in the documentation.



 AlekseyDorkin:

Well, can’t you show the function that you’re passing to map? It’s pretty difficult to figure out what goes wrong otherwise.


It’s a really complicated function, that’s why I didn’t post it, but here it is:
def _aug_map(self, example, aug_function, idx_generator):
        if example["label"][0] != self.minority_class:
            return example
        self.unique_aug = prob_round(self.num_each_sentence_aug)
        if self.unique_aug == 0:
            return example
        sentence = example["sentence"][0]
        augmented_list = pydash.flatten_deep([aug_function.augment(sentence, n=self.unique_aug)])
        label_list = [self.minority_class for _ in range(len(augmented_list))]
        
        idx_list = [next(idx_generator) + i for i in range(len(augmented_list))]

        example["sentence"].extend(augmented_list)
        example["label"].extend(label_list)
        example["idx"].extend(idx_list)
        return example

With lambda, I create another function that takes only example and calls this function with other required arguments, and then I pass that function to map.