We introduce a novel synthetic oversampling method for variable length, multifeature sequence datasets based on autoencoders and generative adversarial networks. We show that this method improves classification accuracy for highly imbalanced sequence classification tasks. We show that this method outperforms standard oversampling techniques that use techniques such as SMOTE and autoencoders. We also use generative adversarial networks on the majority class as an outlier detection method for novelty detection, with limited classification improvement. We show that the use of generative adversarial network based synthetic data improves classification model performance on a variety of sequence data sets.
Dealing with imbalanced datasets is the crux of many real world classification problems. These problems deal with complex multivariate data such as variable length, multi-feature sequence data. Canonical examples can be found in the finance world, for example, questions related to stock market data of several securities or credit card fraud detection often deal with sequence data with many features. Other imbalanced data problems include questions in the medical field such as tumor detection and post surgery prognosis (Zięba et al., 2014). In each of these problems, false positives are more desirable than false negatives, they require sequential data, and the classes are imbalanced.
Class imbalances in datasets oftentimes lead to increased difficulty in classification problems as many machine learning algorithms assume that the dataset is balanced. There are two general approaches to improve classification accuracy for unbalanced datasets. One method is algorithmic, for example, a modified loss function can be used so that misclassifications of minority labeled data are penalized more heavily than misclassifications of majority labeled
This motivates the development of methods to improve classification accuracy on variable length, multi-feature sequence data. Given a sequence of T feature vectors, we want to predict labels of the sequence. Oftentimes it is not obvious how to apply methods for unbalanced data to sequence data in a way that takes advantage of the fact that sequential events have the potential to be highly correlated. SMOTE (Chawla et al., 2002) is widely used for oversampling, but does not capture the sequential dimension. Enhanced Structure Preserved Oversampling (ESPO) (Cao et al., 2013) allows one to generate synthetic data that preserves the sequence structure, however it requires that the feature vector has only a single feature at each of the T time points and that the output label is a scalar. As there is no obvious extension to the case where there are multiple features at each time point and the output is also a sequence of labels, the situations where ESPO can be applied are limited.
We develop a method based on deep learning models for sequences in order to decrease data imbalances of sequence data with an arbitrary number of features. We call each feature vector, x i ∈ R n , an event in the sequence. We consider the use of generative adversarial networks (GANs) to generate synthetic data. Here, we build a generative model that generates both the feature vectors in a sequence as well as the corresponding labels. We benchmark this synthetic data generation technique against a number of models. We demonstrate that the model trained on the GAN based synthetic data outperforms the baseline model, other standard synthetic data generation techniques, and a GAN based novelty detection method. For each of the synthetic data generation methods, we train a sequence-to-sequence model (Sutskever et al., 2014) on the dataset that outputs a sequence with the same length as the label sequence. In addition to benchmarking against existing synthetic data generation techniques, we also train a model on the unaugmented dataset. All of the models are embedded within the standard ensemble approach. On all of our datasets, we observe that the GAN based synthetic data generation model significantly improves over the baseline models by 15% to 127% depending on the dataset, while the novelty detection based GAN performs similarly to the baseline model.
The main contributions are as follows:
1. a novel synthetic data generation technique that uses a GAN with an Autoencoder component to generate synthetic data for variable length, multi-feature sequential data in a way that preserves the structure of sequences for both feature vectors and labels;
2. a new novelty detection method for sequential data that uses a GAN as an outlier detection function;
3. a computational study of existing imbalanced classification techniques on highly imbalanced sequential datasets.
In the next section, we discuss relevant literature. Section 3 discusses all of the models, while the computational results are presented in Section 4.
The paper consider important and interesting problem: how to generate a sequence from minority class if we want to do oversampling with synthetic data in a way similar to SMOTE. 
Now from the paper, the exact used approach is not clear, as the details are too scarce (see some examples below). Experiments are not convincing, as the authors don't compare to the state of the art approaches. As the exact problem statement is hard to grasp, it is hard to identify the exact contribution of authors.
Note, that the proposed approaches, in my opinion, are not that different from approaches from modern data generation for imbalanced classification [1, 2], as we propose some kind of GAN to generate new data and so have only two variables: how we select loss for this GAN and how we select the architecture.
I assume, that to be accepted at a major venue a deeper investigation is required at the moment.
See also the following comments:
1. Figure 4: title is not required, as we have a caption with the same information. Better to use confidence bars too.
2. The figures will benefit from usage of vector format.
3. Figure 3: tSNE can vary from one run to another. It is better to provide at least 3 random figures or even better train e.g. a simple classification model for t-SNE embedded model and present ROC AUC scores for identification synthetic/non-synthetic.
4. Table 1&2 formatting is different from that usually used in Academy (see e.g. https://dl.sciencesocieties.org/files/publications/style/chapter-05.pdf)
5. F1 score is often not the best metric for imbalanced problems. The paper will benefit from providing also PR AUC (average precision) scores.
6. Figure 2: avoid confusion matrices presented in this form, as they take much space providing almost no information. Classic tables are better.
7. From the problem statement at the very beginning of section 2 it is not clear what kind of labels do we expect (I suppose that for each sequence we have a specific label i.e. all y_i are 0 but one, that is 1)
8. Sometimes bigger weights for minority objects or dropping significant part of majority sequences are enough, so results for these approaches also should be included
9. How the hyperparameters mu and lambda were selected?
[1.] Guo, Ting, et al. "Discriminative Sample Generation for Deep Imbalanced Learning." Proceedings of the 28th International Joint Conference on Artificial Intelligence. AAAI Press, 2019. IJCAI 2019, https://www.ijcai.org/proceedings/2019/0334.pdf
[2.] Douzas, Georgios, and Fernando Bacao. "Effective data generation for imbalanced learning using conditional generative adversarial networks." Expert Systems with applications 91 (2018): 464-471.