Learning to Compose Domain-Specific Transformations for Data Augmentation
Data augmentation is a ubiquitous technique for increasing the size of labeled training sets by leveraging task-specific data transformations that preserve class labels. While it is often easy for domain experts to specify individual transformations, constructing and tuning the more sophisticated compositions typically needed to achieve state-of-the-art results is a time-consuming manual task in practice. We propose a method for automating this process by learning a generative sequence model over user-specified transformation functions using a generative adversarial approach. Our method can make use of arbitrary, non-deterministic transformation functions, is robust to misspecified user input, and is trained on unlabeled data. The learned transformation model can then be used to perform data augmentation for any end discriminative model. In our experiments, we show the efficacy of our approach on both image and text datasets, achieving improvements of 4.0 accuracy points on CIFAR-10, 1.4 F1 points on the ACE relation extraction task, and 3.4 accuracy points when using domain-specific transformation operations on a medical imaging dataset as compared to standard heuristic augmentation approaches.
This paper addresses an interesting and new problem to augment training data in a learnable and principled manner. Modern machine learning systems are known for their ‘hunger for data’ and until now state-of-the-art approaches have relied mainly on heuristics to augment labeled training data. This paper tries to reduce the tedious task of finding a good combination of data augmentation strategies with best parameters by learning a sequence of best data augmentation strategies in a generative adversarial framework while working with unsupervised data.
      Plus points:
      1. The motivation behind the problem is to reduce human labor without compromising the final discriminative classification performance. The problem formulation is pretty clear from the text.
      2. The solution strategy is intuitive and based on realistic assumptions. The weak assumption of class invariance under data transformations enable the use of unsupervised training data and a generative adversarial network to learn the optimum composition of transformation functions from a set of predefined data transformations. To address the challenge of transformation functions being possibly non-differentiable and non-deterministic, the authors propose a reinforcement learning based solution to the problem.
      3. The experiments are well designed, well executed and the results are well analyzed. The two variants (mean-field and LSTM) of the proposed method almost always beat the basic and heuristics based baselines. The proposed method shows its merit on 3 different domains with very different application scenarios. The method experimentally proves its adaptability and usefulness of using domain specific transformation functions in the experiment with Mammography Tumor Classification experiment. The authors did a good job by showing the robustness of the method in the TF Misspecification experimentation (L279-286).
      Minus points:
      1. I have confusion regarding the role of the discriminator which I’m writing in detail below. I can understand that the generator's task is to produce less number of out-of-distribution null class (eqn. between L113 and 114). With the weak assumption (eq.(2)), this means that the generator's task is to produce mostly an image of the same class as that of the input. Then if I'm understanding correctly, the discriminator's task is to have low value of D(.) when a transformed example is passed through the discriminator (first term of eqn between L117-L118). At the same time, the discriminator's task is to have high value of D(.) when a non-transformed example is passed through the discriminator (second term of eqn between L117-L118). Thus the 'game' being played between the generator and the discriminator is that the generator is trying to (to propose a sequence of transformations to) generate in-class (i.e., non null-class) examples while the discriminator is trying to tell whether the input coming to it is a null-class (coming from a bad generator which produces null-class example) or not (coming from the real distribution of examples or from a good generator). If my understanding (above) is true, I'd request to put some explanatory texts along this line (the game analogy and what is the role of the discriminator i.e., when will the discriminator give a high value and when low value). And if my understanding is not in the right direction, then I'd like to see a better explanation of what the authors tried to achieve by the generative adversarial objective.
      Side comment: Please provide eqn. numbers to all equations even if it is not referred anywhere in the text. It helps the readers to refer to them.
      In light of the clear motivation of a new problem, succinct solution strategy and well designed and executed experiments I’m positive about the work. However, I’d like to see the authors’ rebuttal about the confusion I wrote above.
=======
After reading the fellow reviewers' comments and the the authors' response in detail, I'm more positive about this work. The authors' did a good job in clarifying my question about the role of the discriminator and have told that they would add a clarification along those lines. I agree with the authors that the focus of the work was on conducting a study of getting optimum sequence of transformation functions for data augmentation and data augmentation makes more sense when the availability of data is limited. Notwithstanding, the authors did a good job by providing the performance of their method fin the fully supervised scenario for the sake of completeness and on the request of the fellow reviewers. This also adds to the robustness of the proposed method. Thus, considering the novel problem definition, neat solution strategy and supportive experiments I am going for an even stronger accept. To me, it is a candidate for an oral presentation.