summaries: The paper presents a novel approach to improve the accuracy of regression models that are learned from a skew dataset. The proposed approach consists of two parts, namely, (i) adversarial network for forcing output distributions and (ii) regularization based on an adversarial autoencoder. Experiments suggest that the proposed approach increases the accuracy of the regression model for all the four datasets considered in the paper.

summaries: This paper proposed to learn a regression model using "skewed data", which is defined as the subset of training samples with true target above certain threshold. The model consists of two components. First, the input x was mapped to its latent space through encoder R_enc. The latent representation was further mapped to the predicted output through regressor network R_post. The predictive distribution was forced to match the true target distribution p(y) through an adversarial network. Second, the latent space representations were also forced to match the true target distribution p(y). Experimental results on synthetic benchmark data showed the proposed approach performed better than naively applying regression model on the skewed data.

summaries: This paper proposed a semi-supervised learning approach to improve the regression model trained on output-skewed data. The key assumption is that, though the training outputs can be skewed, it is easy to estimate the true distribution of the output. The proposed model that combines an AAE that generates the output distribution, and an adversarial model that enforces the distribution of the predicted output to resemble the true distribution of the output. On several real datasets, the ablation study shows the proposed model can improve the regression accuracy.

abstractText: Advances in scientific fields including drug discovery or material design are accompanied by numerous trials and errors. However, generally only representative experimental results are reported. Because of this reporting bias, the distribution of labeled result data can deviate from their true distribution. A regression model can be erroneous if it is built on these skewed data. In this work, we propose a new approach to improve the accuracy of regression models that are trained using a skewed dataset. The method forces the regression outputs to follow the true distribution; the forcing algorithm regularizes the regression results while keeping the information of the training data. We assume the existence of enough unlabeled data that follow the true distribution, and that the true distribution can be roughly estimated from domain knowledge or a few samples. During training neural networks to generate a regression model, an adversarial network is used to force the distribution of predicted values to follow the estimated ‘true’ distribution. We evaluated the proposed approach on four real-world datasets (pLogP, Diamond, House, Elevators). In all four datasets, the proposed approach reduced the root mean squared error of the regression by around 55 percent to 75 percent compared to regression models without adjustment of the distribution.

authors:
J. Alcala-Fdez
A. Fernandez
J. Luengo
J. Derrac
S. Garcia
L. Sanchez
F. Herrera

title: KEEL DataMining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework

venue: Journal of Multiple-Valued Logic and Soft Computing,

authors:
Olivier Chapelle
Bernhard Scholkopf
Alexander Zien

title: Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews

venue: IEEE Transactions on Neural Networks,

authors:
Hailin Chen
Zuping Zhang

title: A semi-supervised method for drug-target interaction prediction with consistency in networks

venue: Plos one,

authors:
Antonia Creswell
Alison Pouplin
Anil A Bharath

title: Denoising adversarial autoencoders: classifying skin lesions using limited labelled training data

venue: IET Computer Vision,

title: Active Learning: an unbiased approach

venue: PhD thesis,

title: Adversarial autoencoders for anomalous event detection in images

authors:
Charles Elkan
Keith Noto

title: Learning classifiers from only positive and unlabeled data

venue: In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining,

authors:
Mikel Galar
Alberto Fernandez
Edurne Barrenechea
Humberto Bustince
Francisco Herrera

title: A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches

venue: IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews),

authors:
Andrew B Goldberg
Xiaojin Zhu

title: Seeing stars when there aren’t many stars: Graph-based semisupervised learning for sentiment categorization. In Proceedings of TextGraphs: The first workshop on graph based methods for natural language

authors:
Yu-Guan Hsieh
Gang Niu
Masashi Sugiyama

title: Classification from positive, unlabeled and biased negative data

venue: In International Conference on Machine Learning,

authors:
Jaehyung Kim
Youngbum Hur
Sejun Park
Eunho Yang
Sung Ju Hwang
Jinwoo Shin

title: Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning

venue: arXiv preprint arXiv:2007.08844,

authors:
Durk P Kingma
Shakir Mohamed
Danilo Jimenez Rezende
Max Welling

title: Semi-supervised learning with deep generative models

venue: In Advances in neural information processing systems,

authors:
Yi Lin
Yoonkyung Lee
Grace Wahba

title: Support vector machines for classification in nonstandard situations

venue: Machine learning,

authors:
Hui Liu
Jianjiang Sun
Jihong Guan
Jie Zheng
Shuigeng Zhou

title: Improving compound–protein interaction prediction by building up highly credible negative samples

venue: Bioinformatics, 31:i221––i229,

authors:
Turab Lookman
Prasanna V. Balachandran
Dezhen Xue
Ruihao Yuan

title: Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design

venue: npj Computational Materials,

authors:
Lihong Peng
Wen Zhu
Bo Liao
Yu Duan
Min Chen
Yi Chen
Jialiang Yang

title: Screening drug-target interactions with positive-unlabeled learning

venue: Scientific Reports,

authors:
Daniil Polykovskiy
Alexander Zhebrak
Benjamin Sanchez-Lengeling
Sergey Golovanov
Oktai Tatanov
Stanislav Belyaev
Rauf Kurbanov
Aleksey Artamonov
Vladimir Aladinskiy
Mark Veselov
Artur Kadurin
Simon Johansson
Hongming Chen
Sergey Nikolenko
Alan Aspuru-Guzik
Alex Zhavoronkov

title: Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models

venue: arXiv preprint arXiv:1811.12823,

authors:
Ameya Prabhu
Charles Dognin
Maneesh Singh

title: Sampling bias in deep active classification: An empirical study

venue: arXiv preprint arXiv:1909.09389,

authors:
Mehdi Rezagholiradeh
Md Akmal Haidar

title: Reg-gan: Semi-supervised learning based on generative adversarial networks for regression

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Chuck Rosenberg
Martial Hebert
Henry Schneiderman

title: Semi-supervised self-training of object detection models

authors:
Bertrand Rouet-Leduc
Kipton Barros
Turab Lookman
Colin J. Humphreys

title: Optimisation of GaN LEDs and the reduction of efficiency droop using active machine learning

authors:
Kihyuk Sohn
David Berthelot
Chun-Liang Li
Zizhao Zhang
Nicholas Carlini
Ekin D Cubuk
Alex Kurakin
Han Zhang
Colin Raffel

title: Fixmatch: Simplifying semi-supervised learning with consistency and confidence

authors:
Luı́s Torgo
Rita P Ribeiro
Bernhard Pfahringer
Paula Branco

title: Smote for regression

venue: In Portuguese conference on artificial intelligence,

title: ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016

authors:
Yutong Xie
Jianpeng Zhang
Yong Xia

title: Semi-supervised adversarial model for benign–malignant lung nodule classification on chest ct

venue: Medical image analysis,

authors:
Noah Youngs
Dennis Shasha
Richard Bonneau

title: Positive-unlabeled learning in the face of labeling bias

venue: In 2015 IEEE International Conference on Data Mining Workshop (ICDMW),

authors:
Ruihao Yuan
Zhen Liu
Prasanna V. Balachandran
Deqing Xue
Yumei Zhou
Xiangdong Ding
Jun Sun
Dezhen Xue
Turab Lookman

title: Accelerated Discovery of Large Electrostrains in BaTiO3-Based Piezoelectrics Using Active Learning

venue: Advanced Materials,

authors:
Xiaojin Zhu
Zoubin Ghahramani
John D Lafferty

title: Semi-supervised learning using gaussian fields and harmonic functions

venue: In Proceedings of the 20th International conference on Machine learning

heading: 1 INTRODUCTION

text:
Advances in scientific fields including drug discovery or material design are accompanied by numerous trials and errors. However, generally only representative experimental results are chosen to be reported. As a consequence of this reporting bias, the distribution of the reported results can differ from the true distribution. For this reason, when data from the literature are used to train a regression model, predictions from the regression model may differ from the true distribution because the model is derived using biased data (Lin et al., 2002; Galar et al., 2011).
In particular, pharmaceutical development is often affected by this problem. Quantitative structure-activity relationship (QSAR), including drug-target interaction (DTI), is consistently affected by the bias in the reported experimental data, because usually the targeted range of molecular property is clearly defined (Liu et al., 2015; Chen & Zhang, 2013). When regression is performed using such skewed data, it often erroneously predicts that the target properties are satisfied. As a consequence, it is difficult to discover molecules that have the desired properties (Peng et al., 2017).
Active learning applications also have a similar problem. Many active learning methods repeat the selection of new data by applying certain criteria and retraining the surrogate model (Lookman et al., 2019; RouetLeduc et al., 2016; Yuan et al., 2018). During this process, the data can be skewed according to the criteria
(de Mello, 2013; Prabhu et al., 2019). However, despite this problem, few studies have tried to improve the accuracy of regression models that have been trained on skewed data.
In this work, we propose a new approach to improve the accuracy of a regression model that is trained using skewed data. We assume the presence of enough unlabeled data which follow the true distribution, and that the true distribution can be roughly estimated using domain knowledge or a few examples. We use a semi-supervised learning framework with an adversarial network to force the distribution of the regression output to resemble the assumed true distribution (Figure 1). At the same time, by sharing the front part of the regression model with the encoder of an adversarial autoencoder (AAE), the process of forcing the distribution of output values is regularized in a way that the information of the labeled data is represented stably. We created skewed datasets by selecting data that exceeded a certain threshold from four real-world datasets (pLogP, Diamond, House, Elevators), then evaluated the proposed approach using these skewed datasets. The proposed approach reduced the root mean squared error (RMSE) of the regression model derived using each of the four datasets, compared to the regression model that had been trained using only the skewed datasets. We also verified that the proposed approach is feasible even when the estimate of the true distribution is not perfect.

heading: 2 RELATED WORK

text:
Semi-Supervised Learning (SSL) is a machine-learning strategy to learn using partially-labeled datasets (Chapelle et al., 2009). In the field of SSL, various methods have been developed, including those using generative models (Kingma et al., 2014), graphs (Goldberg & Zhu, 2006), self-training (Rosenberg et al., 2005) and consistency regularization (Sohn et al., 2020). SSL can improve classification and regression models by using information in a large set of unlabeled data to train a relatively small set of labeled data (Xie et al., 2019; Creswell et al., 2018; Dimokranitou, 2017; Rezagholiradeh & Haidar, 2018). These approaches generally assume that labeled and unlabeled datasets are well distributed without distortion.
Several methods were tried to train a model on data that have biased distribution. A method proposed by (Zhu et al., 2003) incorporated class prior knowledge to adjust the class distribution. Another method proposed by (Kim et al., 2020) used a model trained with skewed labeled data to generate pseudo-labels of unlabeled data for retraining by considering the entropy of prediction. Positive Unlabeled (PU) learning uses data with only positive and unlabeled samples for binary classifications (Elkan & Noto, 2008); PU learning has been studied intensively (Hsieh et al., 2019; Youngs et al., 2015). However, to the best of our knowledge, few methods used skewed labeled datasets for regression.
Although other methods for learning with imbalanced data have been developed, such as the Synthetic Minority Oversampling Technique (SMOTE) for regression (Torgo et al., 2013), these approaches can only be applied if even some of the data are in the sparse area.

heading: 3 METHOD

text:
Our proposed approach is aimed to improve accuracy of regression models that are trained on a dataset that is skewed according to labels. Dl denotes a skewed labeled dataset with n samples that satisfies
yi > θ, ∀(xi, yi) ∈ Dl = {(xi, yi)}ni=1 (1)
where xi ∈ Rd is d-dimensional input, yi ∈ R is ground truth target and θ is the bias threshold above which the labeled data are present. We assume that we have an estimate of the true distribution of targets y, p(y), and that we have an unbiased unlabeled dataset Du = {xn+1,xn+2, . . . ,xm} that has sufficiently large m n and in which the distribution of ground truth targets follows p(y). In this section, we describe our approach to make the regression outputs follow the true distribution as closely as possible by using Dl and Du in semi-supervised fashion. The proposed approach consists of two parts: “Adversarial network for forcing output distribution” and “Regularization using AAE”. In the first part, we present a method that uses an adversarial network to force the output distribution of the regression model to be similar to p(y). Then, we describe a method to properly regularize this forcing process by implementing an AAE.

heading: 3.1 ADVERSARIAL NETWORK FOR FORCING OUTPUT DISTRIBUTION

text:
Let ŷ be the output of a regressor that aims to predict ground truth targets y, and q(ŷ|x) be the regression distribution. Then for given data distribution pd(x), a distribution of predicted regressor output values q(ŷ) can be defined as:
q(ŷ) = ∫ x q(ŷ|x)pd(x)dx (2)
We use ŷu to denote the predicted values for Du, and ŷl to denote the predicted values for Dl. Then the distributions of predicted values are denoted as q(ŷu) and q(ŷl).
The proposed approach is to regularize the regressor to have q(ŷu) close to p(y). To do so, an adversarial network is placed at the end of the regressor (Figure 1). The model is concurrently trained on Dl for regression, and on Du to force the regression outputs to have similar distribution to p(y). The discriminator distinguishes outputs of the regressor from randomly-sampled values in p(y), and the regressor learns to deceive the discriminator.
Suppose we divide Du into two subsets, Dabove that consists of x that have ground truth target values > θ, and Dbelow which consist of x that have ground truth target values < θ. If the regressor is well trained
on Dl, then prediction on Dabove is expected to be relatively accurate because the distributions of x of Dl and Dabove are similar. In contrast, for Dbelow, the regressor should predict values that have not ever been seen; therefore, we hypothesized that during adversarial training on Du, the prediction on Dabove should stay relatively still, whereas prediction on Dbelow is flexible. If this hypothesis holds, then the portion of q(ŷu) that is > θ should be filled with prediction outputs on Dabove, and prediction outputs on Dbelow should be < θ to deceive the discriminator. These responses result in the regressor having q(ŷu) close to p(y) while maintaining accurate prediction for outputs > θ.

heading: 3.2 REGULARIZATION BASED ON AAE

text:
Even if prediction on Dabove is intact and the predicted output distribution of Du fits the assumed true distribution, the prediction on Dbelow will still have a large error in most cases. To guide the prediction output on Dbelow correctly, the information from above θ must be conveyed to below θ. We used the AAE to regularize the process of forcing the output distribution to achieve appropriate propagation of the information.
We set the encoder of an AAE to share the front part of the regression network (Figure 1). We denote the shared part of the regression network as Renc and rest of the network as Rpost. Renc works as the encoder of the AAE and the front part of the regression network at the same time. Therefore, latent vectors of the AAE should act as a useful feature for the Rpost which functions to predict y from the latent vectors. To fulfil its function, Renc must consider knowledge from the labeled dataset.
Also, during the process of forcing the output distribution,Rpost also transforms the given distribution of the latent vectors to p(y). To be a useful feature for Rpost, latent vectors should be arranged in a similar way to p(y), which possesses information about how the labeled dataset is skewed. This implies that latent vectors become more useful if Renc is aware of the skewed distribution of the labeled dataset. To facilitate this process, we shaped the distribution of the latent vectors to have similar characteristics to p(y) by adjusting the target distribution of AAE.
Consequently, we hypothesized that the regularization process of AAE can be well guided by this information from labeled data and p(y). This good guidance results in well-controlled propagation of information from the portion > θ to the prediction on Dbelow.

heading: 4 EXPERIMENTS

text:
In this section, we present results of experiments on regression of artificially-skewed datasets. For a given dataset, we first randomly sampled a sufficient amount of data to construct an unlabeled train dataset. Then we divided the rest of the data arbitrarily into three datasets to form a labeled train dataset, a validation dataset and a test dataset. To induce bias into the labeled dataset fairly, only data with labels that had values more than one standard deviation above the mean of all labels were selected randomly until the labeled dataset was sufficiently large.
For the experiments, we used four datasets: moses (QSAR on calculated pLogP), diamond, house and elevators (Polykovskiy et al., 2018; Wickham, 2016; Alcala-Fdez et al., 2011). Since QSAR on small molecules suffers from skewed datasets in many practical applications, our experiments are focused on the QSAR on calculated pLogP task. Other datasets were used to demonstrate the general applicability of the proposed approach.
We focused on comparing regression accuracy of the models that were trained using our approach or without it. Further, to explore the effectiveness at which the adversarial network forced output distribution and AAE based regularization separately, we conducted ablation studies. As the proposed approach requires an estimate of the true distribution, we tested the sensitivity of the model to the quality of the estimate. In each test, the true distribution of the data was estimated to be Gaussian distribution that has a mean and standard
deviation of 1) full data or 2) 20 randomly-sampled data. We set the assumed true distribution as Gaussian distribution for experiments on all four datasets regardless of their real distributions.
These experiments should not consider the label of unlabeled dataset until the final accuracy test. Therefore, to tune hyperparameters, we only considered accuracy of the regression on labeled dataset and distribution mismatch between predicted values and the assumed true distribution.

heading: 4.1 EXPERIMENTS ON QSAR TASK

text: The agreement between ground truth and predicted pLogP were affected by the method used. The model trained using fully-labeled data accurately predicted pLogP (Figure 2a). However, the model trained with skewed data without the proposed approach, showed decrease in accuracy as the ground truth value declined below the bias threshold (Figure 2b); this result indicates that the model’s extrapolation ability is not sufficient for this task. In contrast, the model trained with the proposed approach followed the data well (Figure 2c). The RMSE (Table 1) also show that about 70% of error in the model without the proposed approach is reduced using the proposed approach. This result suggests that information learned from skewed labeled data can guide the outputs of the regression model to properly fit in the assumed true distribution during the training process; otherwise, the regression model with proposed approach would show predictions that have a distribution that matches the assumed true distribution, but would not be accurate.

heading: 4.2 ABLATION STUDY

text:
To address the separate influence of the adversarial network to force output distribution and AAE for regularization, we conducted an ablation study that excluded each component.
When only the forcing output distribution part was excluded, the model showed almost uniform values outside of the labeled region (Figure 6a). The result indicates that regularization alone is not sufficient for extrapolation in this task. Comparison with (Figure 2b) demonstrates that the regularization of the predicted values reduces the variation, but that the effect of skewed dataset holds the predicted values to near the lowest labeled data point. As a result, the RMSE of the model is not meaningfully improved from that of the model without the proposed approach (Table 1).
When only AAE for regularization was excluded, the prediction values were relatively accurate but the trend of prediction according to ground truth fluctuated slightly (Figure 6b). The fluctuation suggests a decrease in the ability to convey information from labeled data. Although the RMSE of the regression model was
∼40% lower than that of the model without the proposed approach (Table 1), strong regularization seems wise, because in practice, we cannot evaluate the error of the model if fully-labeled data are not available.

heading: 4.3 GENERAL APPLICABILITY OF PROPOSED APPROACH

text:
RMSE were also collected for the models for datasets other than QSAR (Table 1). For the other three datasets, RMSE was reduced by 55% to 75% compared to the RMSE of the regression model trained with the skewed dataset and without the proposed approach. The result was consistent even for the difficult task that shows high RMSE (e.g., house). This result demonstrates that even slight information from the labeled data may guide the process of forcing output distribution.
However, in practice, tasks may be performed on datasets for which the true distribution cannot be estimated accurately even when the dataset is known to be skewed. The proposed approach relies strongly on the assumed true distribution, so we tested sensitivity of the proposed approach to the assumed true distribution. For this purpose, we estimated the true distribution of the datasets by using only 20 random samples instead of the full data. Interestingly, the degree of RMSE improvement over the model without the proposed approach was reduced by only 10% to 20%. Considering that the true distributions of the datasets are not exactly Gaussian, this result indicates that the proposed approach is not sensitive to error in the assumed true distribution, and is appropriate for practical applications.

heading: 5 CONCLUSION

text: We proposed a new approach to improve a regression model that is trained on skewed dataset. The method uses adversarial forcing to make the output distribution follow the assumed true distribution. The adversarial network to force the output distribution restrains a regression model to have the same distribution as the output distribution, and the AAE regularizes the model in the appropriate way. We evaluated the proposed approach on four datasets. The proposed approach increased the accuracy of the regression model on all datasets. Further assessment showed that the regression quality is maintained even when the proposed approach is applied with rough estimation of the true distribution.

heading: A APPENDIX

text: A.1 EXPERIMENTS ON OTHER TYPES OF BIASES

title: SEMI-SUPERVISED REGRESSION WITH SKEWED DATA VIA ADVERSARIALLY FORCING THE DISTRIBUTION OF PREDICTED VALUES


summaries: This paper proposes an architecture that addresses transferability of compositionality. The proposed architecture consists of three components: a network that transforms the input X into a series of hidden representations {H_1, H_2, ... H_K}, a network that reconstructs the input X from this series of hidden representations, and a prediction network that generates a prediction from the hidden representations. The authors propose several datasets meant to address transferability of compositional generalisation, and show that their architecture significantly improves standard DNN architectures as well as humans on these datasets.

summaries: The paper introduces a “transferability of compositionality” problem and proposes an approach to alleviate it. The said problem may arise when one trains neural models to produce “compositional” representations of the input. In the paper “compositional representations” consist of multiple vectors which are supposed to correspond to semantically meaningful aspects of the input, for example different objects in the case of images or different parts of compound words in the case of linguistic inputs. The transferability problem arises when there is a difference between training and test distributions, namely when certain combinations of objects have different probabilities in training & testing. The proposed solution at inference time is to project object representations to the manifold of individual object representations. The manifold is estimated by saving representations of individual object representations from the training time.

summaries: This paper studies "compositionality" and in particular the way in which it "transfers" on test data. They run simple baselines on three experiments (overlapped MNIST, colored MNIST and concatenated month names) and find that the baselines do not learn compositional representation. They proposed the use of an *auxiliary reconstruction network and a regularized optimization* which improves on these baselines.

abstractText: Compositional generalization is the algebraic capacity to understand and produce large amount of novel combinations from known components. It is a key element of human intelligence for out-of-distribution generalization. To equip neural networks with such ability, many algorithms have been proposed to extract compositional representations from the training distribution. However, it has not been discussed whether the trained model can still extract such representations in the test distribution. In this paper, we argue that the extraction ability does not transfer naturally, because the extraction network suffers from the divergence of distributions. To address this problem, we propose to use an auxiliary reconstruction network with regularized hidden representations as input, and optimize the representations during inference. The proposed approach significantly improves accuracy, showing more than a 20% absolute increase in various experiments compared with baselines. To our best knowledge, this is the first work to focus on the transferability of compositionality, and it is orthogonal to existing efforts of learning compositional representations in training distribution. We hope this work will help to advance compositional generalization and artificial intelligence research. The code is in supplementary materials.

authors:
Joost Bastings
Marco Baroni
Jason Weston
Kyunghyun Cho
Douwe Kiela

title: Jump to better conclusions: Scan both left and right

title: Deep learning of representations: Looking forward

venue: In International Conference on Statistical Language and Speech Processing,

title: The consciousness prior

venue: arXiv preprint arXiv:1709.08568,

authors:
Philémon Brakel
Stefan Frank

title: Strong systematicity in sentence processing by simple recurrent networks

venue: In 31th Annual Conference of the Cognitive Science Society

authors:
Christopher P Burgess
Irina Higgins
Arka Pal
Loic Matthey
Nick Watters
Guillaume Desjardins
Alexander Lerchner

title: Understanding disentangling in β-vae

venue: arXiv preprint arXiv:1804.03599,

authors:
Tian Qi Chen
Xuechen Li
Roger B Grosse
David K Duvenaud

title: Isolating sources of disentanglement in variational autoencoders

venue: In Advances in Neural Information Processing Systems,

authors:
Jerry A Fodor
Ernest Lepore

title: The compositionality papers

authors:
Jerry A Fodor
Zenon W

title: Pylyshyn. Connectionism and cognitive architecture: A critical analysis

authors:
J. Gama
I. Žliobaitė
A. Bifet
M. Pechenizkiy
A. Bouchachia

title: A survey on concept drift adaptation

venue: ACM computing surveys (CSUR),

authors:
J. Gorden
D. Lopez-Paz
M. Baroni
D. Bouchacourt

title: Permutation eqivariant models for compositional generalization in language

venue: In ICLR,

title: Scan: Learning hierarchical compositional visual concepts

authors:
Irina Higgins
Loic Matthey
Arka Pal
Christopher Burgess
Xavier Glorot
Matthew Botvinick
Shakir Mohamed
Alexander Lerchner

title: β-vae: Learning basic visual concepts with a constrained variational framework

venue: In International Conference on Learning Representations (ICLR),

authors:
Jun-Ting Hsieh
Bingbin Liu
De-An Huang
Li F Fei-Fei
Juan Carlos Niebles

title: Learning to decompose and disentangle representations for video prediction

title: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

title: Measureing compositional generalization: a comprehensive method on realistic data

authors:
Hyunjik Kim
Andriy Mnih

title: Disentangling by factorising

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Markus Kliegl
Wei Xu

title: More systematic than claimed

venue: Insights on the scan tasks. OpenReview,

authors:
Abhishek Kumar
Prasanna Sattigeri
Avinash Balakrishnan

title: Variational inference of disentangled latent concepts from unlabeled observations

authors:
Brenden Lake
Marco Baroni

title: Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

title: Compositional generalization through meta sequence-to-sequence learning

authors:
Brenden M Lake
Tomer D Ullman
Joshua B Tenenbaum
Samuel J Gershman

title: Building machines that learn and think like people

venue: Behavioral and Brain Sciences,

authors:
Brenden M Lake
Tal Linzen
Marco Baroni

title: Human few-shot learning of compositional instructions

venue: arXiv preprint arXiv:1901.04587,

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Yuanpeng Li
Liang Zhao
Jianyu Wang
Joel Hestness

title: Compositional generalization for primitive substitutions

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),

authors:
Tal Linzen
Emmanuel Dupoux
Yoav Goldberg

title: Assessing the ability of lstms to learn syntaxsensitive dependencies

venue: Transactions of the Association for Computational Linguistics,

authors:
Francesco Locatello
Stefan Bauer
Mario Lucic
Gunnar Raetsch
Sylvain Gelly
Bernhard Schölkopf
Olivier Bachem

title: Challenging common assumptions in the unsupervised learning of disentangled representations

authors:
Francesco Locatello
Michael Tschannen
Stefan Bauer
Gunnar Ratsch
Bernhard Scholkopf
Olivier Bachem

title: Disentangling factors of variations using few labels

authors:
Joao Loula
Marco Baroni
Brenden M Lake

title: Rearranging the familiar: Testing compositional generalization in recurrent networks

venue: arXiv preprint arXiv:1807.07545,

authors:
Laurens van der Maaten
Geoffrey Hinton

title: Visualizing data using t-sne

venue: Journal of machine learning research,

title: Rethinking eliminative connectionism

venue: Cognitive psychology,

title: The algebraic mind: Integrating connectionism and cognitive science

venue: MIT press,

authors:
Geoff Pleiss
Amauri Souza
Joseph Kim
Boyi Li
Kilian Q. Weinberger

title: Neural network out-of-distribution detection for regression tasks, 2020. URL https://openreview.net/ forum?id=ryxsUySFwr

authors:
I. Redko
E. Morvant
A. Habrard
M. Sebban

title: Bennanl. A survey on domain adaptation theory

venue: arXiv preprint arXiv:2004.11829v1,

authors:
Paul Rodriguez
Janet Wiles

title: Recurrent neural networks can learn to implement symbol-sensitive counting

authors:
Jake Russin
Jason Jo
Randall C O’Reilly

title: Compositional generalization in a deep seq2seq model by separating syntax and semantics

authors:
Gail Weiss
Yoav Goldberg
Eran Yahav

title: On the practical computational power of finite precision rnns for language recognition

venue: arXiv preprint arXiv:1805.04908,

authors:
Francis CK Wong
William SY Wang

title: Generalisation towards combinatorial productivity in language acquisition by simple recurrent networks

venue: In 2007 International Conference on Integration of Knowledge Intensive Multi-Agent Systems,

authors:
Guangyu Robert Yang
Madhura R Joglekar
H Francis Song
William T Newsome
Xiao-Jing Wang

title: Task representations in neural networks trained to perform many cognitive tasks

venue: Nature neuroscience,

heading: 1 INTRODUCTION

text:
Human intelligence (Minsky, 1986; Lake et al., 2017) exhibits compositional generalization, the algebraic capacity to understand and produce large amount of novel combinations from known components (Chomsky, 1957; Montague, 1970). This capacity helps humans to recognize the world efficiently and to be imaginative. It is also beneficial to design machine learning algorithms with compositional generalization skills. Current neural network models, however, generally lack such ability. Compositional generalization is a type of out-of-distribution generalization (Bengio, 2017), where the training and test distributions are different. A sample in such a setting is a combination of several components, and the generalization is enabled by recombining the seen components of the unseen combination during inference. In the image domain, an object is a combination of many parts or properties. In the language domain, a compound word is a combination of multiple words. As an example, we consider two digits are overlapped (Figure 1). Each digit is a component, and it appears in training. A test example has a new combination of two digits.
The main approach for compositional generalization is to learn compositional representations (Bengio, 2013), which contain several component representations. Each of them depends only on the underlying generative factor, and does not change when other factors change. We call this the compositionality property, and will formally introduce in Section 3. In the digit example, this means that the representation of one digit does not change when the other digit changes.
Multiple approaches have been proposed to learn compositional representations in the train distribution. However, little discussion has focused on whether the model can still extract the representations in the test distribution. We find that the extraction ability does not transfer naturally, because the extraction network suffers from the divergence of distributions (Bengio, 2017; Pleiss et al., 2020), so that each extracted representation shifts away from the corresponding one in training. Our experiment on the digit example shows that the accuracy drops from 89.6% in training to 49.3% in test (Table 1 in Section 5).
To address the problem, we hope each representation is consistent with the training one while reflecting the test sample. We use an auxiliary network, which has hidden representations as inputs,
and the original input as output. For a test sample, we regularize each hidden representation in its training manifold, and optimize them to recover the original input. Then we use the optimized representations for prediction. Experimental results show that the proposed approach has more than a 20% absolute increase in various experiments compared to baselines, and even outperforms humans on the overlapping digit task. Our contributions can be summarized as follows.
• We raise and investigate the problem of transferability of compositionality to test distribution. This work is orthogonal to many efforts of learning compositionality in training distribution.
• We propose to address the problem by using an auxiliary reconstruction network with regularized hidden representations as input, and optimize the representations during inference.
• We empirically show that the transferability problem exists and the proposed approach has significant improvements over baselines.

heading: 2 RELATED WORK

text:
Compositional generalization (Chomsky, 1957; Montague, 1970) is critical in human cognition (Minsky, 1986; Lake et al., 2017; Johnson & et al, 2017; Higgins & et al, 2018; Lake et al., 2019). It helps humans to understand and produce large amount of novel combinations from known components. Broadly speaking, compositional generalization is a type of out-of-distribution (o.o.d.) transferring or generalization, which is also called domain adaptation (Redko et al., 2020) or concept drift (Gama et al., 2014). This is different from traditional i.i.d. setting, where the training and the test distributions are identical. The transferring requires prior knowledge of how the distribution is changed, and compositional generalization has a particular form of such change, as mentioned in the later section.
Compositional generalization is also a desirable property for deep neural networks. Human-level compositional learning (Marcus, 2003; Lake & Baroni, 2018) has been an important open challenge (Yang et al., 2019; Keysers & et al, 2020), although there is a long history of studying compositionality in neural networks. Classic view (Fodor & Pylyshyn, 1988; Marcus, 1998; Fodor & Lepore, 2002) considers conventional neural networks lack systematic compositionality. With the breakthroughs in deep neural networks, there are more contemporary attempts to encode compositionality in deep neural networks. Compositionality in neural networks is actively explored for systematic behaviour (Wong & Wang, 2007; Brakel & Frank, 2009), counting ability (Rodriguez & Wiles, 1998; Weiss et al., 2018) and sensitivity to hierarchical structure (Linzen et al., 2016). Researchers have also proposed multiple related tasks (Lake & Baroni, 2018; Loula et al., 2018; Lake et al., 2019) and methods (Lake et al., 2017; Lake & Baroni, 2018; Loula et al., 2018; Kliegl & Xu, 2018; Li et al., 2019; Lake, 2019; Gorden et al., 2020) for learning compositionality in training distribution. Another line of related work is independent disentangled representation learning (Higgins et al., 2017; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017; Hsieh et al., 2018; Locatello et al., 2019; 2020). Its main assumption is that the expected components are statistically independent in training
data. This setting does not have transferring problem in test, because all combinations have positive joint probabilities in training (please see Section 3).
More recently, there have been approaches for better compositional learning in NLP tasks by elaborating RNN models (Bastings et al., 2018), by using pointer networks (Kliegl & Xu, 2018), or by using two representations of a sentence (Russin et al., 2019; Li et al., 2019). In these tasks, the input can be divided into words, which have consistent information in different distributions. However, such property is not always available, e.g. the overlapping digits. In this paper, we propose to acquire the compositional representations with optimization during inference. We will discuss more in the following sections.

heading: 3 COMPOSITIONALITY AND TRANSFERABILITY

text:
In this section, we describe compositionality as a model property. We then argue that compositionality may not transfer to test distribution, and discuss ideas of the proposed approach.
Compositionality Compositional generalization has different training and test distributions. Samples in both training and test data are combinations of K components. For example, in Figure 1, input X has two digits X1, X2, and output Y has corresponding labels Y1, Y2. While a test sample’s combination does not appear in training, each component of the test sample appears in training.
A key for compositional generalization is to have compositional representation, which has multiple component representations. Each component representation corresponds to an underlying input component, and it does not change when other components change. We call this property as compositionality. A compositional representation is computed from an entangled input, and the extraction network needs to output correct component representations. If the extraction network transfers to test distribution, the representations can be correctly extracted in test, so that the compositional generalization is enabled by recombining them.
Transferability As mentioned above, to enable compositional generalization, there is an assumption that the property of compositionality should transfer to the test distribution. Since X1, . . . , XK are entangled, the model has the entire X as input. However, input X has different distributions in training and test, so the extraction network suffers from the divergence of distributions (Bengio, 2017; Pleiss et al., 2020). Hence, even if the model is trained to fit the compositionality property in the training distribution, the property is not guaranteed to transfer to the test distribution.
We propose to obtain compositional representations not from the extractor but reversely from an auxiliary network. We extract compositional representations with optimization, and introduce regularization to make each hidden representation in the corresponding training manifold. We then use the optimized hidden representation for prediction. More details are provided in the next section.

heading: 4 APPROACH

text: In this section, we introduce the proposed approach from model architecture, training and inference perspectives. The architecture contains three modules, trained with routine end-to-end optimization. The inference, different from conventional procedure, includes three steps: extracts initial hidden representations; optimizes hidden representations as module input; predicts output. Figure 2 contains overall flowcharts, and Algorithm 1 is a summary for the approach. We describe details here.

heading: 4.1 MODEL ARCHITECTURE

text:
The model takes a sample with input X and label Y . We have a representation extractor g with parameter φ, which takes X as input, and outputs K hidden representations H = H1, . . . ,HK : H = g(X;φ). We also have a prediction network f with parameter θ, which takes hidden representations H as input, and outputs Ŷ : Ŷ = f(H; θ). These networks can be some existing networks for compositionality learning. In addition to them, we have an auxiliary network h with parameters ψ, which takes hidden representations H as input, and combines them to output X̂: X̂ = h(H;ψ).
Algorithm 1 The proposed approach for training (left) and inference (right). α, β, γ, η are hyper parameters. K is the number of components. M is the number of instances in memory.
Training sample: X,Y 1: H = H1, . . . ,HK = g(X;φ) 2: H ′ = H + η , ∈ N (0, I) 3: Ŷ = f(H ′; θ), X̂ = h(H ′;ψ) 4: Ltrain = CE(Y, Ŷ ) +αL2(X, X̂) + βL2(H) 5: θ̂, φ̂, ψ̂ = argminθ,φ,ψ Ltrain(X,Y, θ, φ, ψ) 6: Memm = g(Xm; φ̂),m = 1, . . . ,M Inference sample: X 1: H init = g(X; φ̂) 2: X̂ = h(H; ψ̂) 3: Lmanf(H) = ∑K k=1 minm L2(Hk,Mem m k ) 4: Linfer(X,H) = L2(X, X̂) + γLmanf(H) 5: Ĥ = argminH Linfer(X,H), H0 = H init 6: Ŷ = f(Ĥ; θ̂)

heading: 4.2 TRAINING

text:
In training, we sequentially use the extractor g and predictor f , by setting the output of the extractor H as input of the predictor. We have a loss Loriginal (containing regularization terms) , such as cross entropy: CE(Y, Ŷ ), to train a model with compositionality by existing algorithms.
On top of that, we train ψ with H as inputs and X̂ as output. We set auxiliary loss as the difference (L2 distance) between X and X̂: Lauxiliary = L2(X, X̂). We also regularize the L2 norm of H , Lhidden = L2(H), and add noise, H ′ = H + η , ∈ N (0, I), to avoid remembering X . η is a hyper parameter. The whole train loss Ltrain is the combination of the original loss, auxiliary loss Lauxiliary, and regularization Lhidden, with coefficients α, β.
Ltrain = Loriginal + αLauxiliary + βLhidden
We train the model in end-to-end manner. This can be a standard training for neural networks.
θ̂, φ̂, ψ̂ = argmin θ,φ,ψ
Ltrain(X,Y, θ, φ, ψ)
After training, we store hidden representations for M training samples. They are used to restrict test representation manifold during inference.
Memm = g(Xm; φ̂), ∀m = 1, . . . ,M

heading: 4.3 INFERENCE

text:
We use optimization to acquire hidden representations during inference. Given a test sample X , the model predicts its output Ŷ . We use the auxiliary network h(H; ψ̂) to search for hidden representations H so that h can output X̂ that is close to the original intput X . This can be achieved by optimization on H with auxiliary loss Lauxiliary. The initial value H init is the output of extractor g: H init = g(X; φ̂).
We also add a manifold regularization term Lmanf to constrain each hidden representation to lie in the corresponding training manifold. For a test sample, we compute the minimum of L2 distance between each of its hidden representation Hk and the corresponding representations in memory Memmk . We
then use the sum of the distances as regularization.
Lmanf(H) = K∑ k=1 min m=1,...,M L2(Hk,Memmk )
Inference loss Linfer is the combination of auxiliary loss and the regularization, with γ as a coefficient.
Linfer(X,H) = Lauxiliary(X, X̂) + γLmanf(H)
Then, we obtain the hidden representations by optimization.
Ĥ = argmin H
Linfer(X,H), H0 = H init
We get prediction from the optimized hidden representations: Ŷ = f(Ĥ; θ̂).

heading: 5 EXPERIMENTS

text:
In this section, we show examples that, given a model with compositionality in training distribution, the compositionality does not transfer to test distributions, and how the proposed approach is applied to these cases. As our focus is orthogonal with learning compositional model in training distribution, we obtain this model by directly providing true labels for each compositional component during training. Then we evaluate the transferability to test distribution.
Since this is the first work for the transferability problem of compositionality, we do not have previous baselines, so that we use standard deep neural network models as baselines, and also compare with variations of the proposed approach. The main changes of the proposed approach include using noise, auxiliary network and manifold regularization. A variation removes one of these changes, so it is also an ablation experiment. The details of hyper parameters and experiment settings can be found in Appendix A. For all the experiments, we use accuracy as metric. A prediction is correct if and only if all the components are correctly predicted. We repeat each experiment five times and report mean and variance.

heading: 5.1 EXPERIMENTS ON OVERLAPPING DIGITS

text:
The first experiment is on overlapping hand written digit recognition, as shown in Figure 1. We construct the dataset from MNIST (LeCun et al., 1998). A sample is made by overlapping and taking the average of two original images, the first one in its original form, and the second flipped over up-left to down-right diagonal. The output is a vector of the two labels Y = Y1, Y2 (not exchangeable). Each original label has 10 possible values, so the output has 100 possible values. To evaluate compositional generalization, we use different distributions in training and test. In train, the sum of the two labels is even, i.e. (Y1 + Y2) mod 2 = 0. In test, the sum is odd.
As a baseline, we use a standard neural network with two sub networks, each for an output. Each of the network is a three layer convolutional neural network. We train the model with cross entropy on both outputs.
The proposed method uses an auxiliary network that takes hidden layer as input and outputs the reconstruction of the original input. The auxiliary network has two sub-networks each with one
Input Outputs Hidden januarymarch 0, 2 january march februaryoctober 1, 9 february october januaryfebruary 0, 1 january february augustmay 7, 4 august may
Table 2: Examples of compound word experiments. The output labels align with corresponding hidden words. Upper is train, lower is test.
hidden representation as input. Each sub-network is a three layer trans-convolutional neural network, and we average the outputs to recover the original input. We use L2 loss as the auxiliary loss.
We also collect human performance data through crowd sourcing. There are 27 participants, and each person works on 20 fixed samples randomly selected from test data. Please refer to Appendix B for more details.
The results in Table 1 (left) show that the proposed method has significant improvement over the baseline by about 20% absolute increase, and it also outperforms humans. The ablation study shows that performances drop in the experiments, indicating that all the modifications in the proposed approach are necessary to achieve the result.

heading: 5.2 EXPERIMENTS ON COMPOUND WORDS

text:
We also conduct experiments for language processing. Language has natural units of words, which have consistent information across different distributions. So we design a setting that we cannot use this property. We consider a problem that converts a compound word to two words. We construct compound words from two month names (January to October), e.g. “julyfebruary”. The output label is the zero-based index of the month (0 for January). We have each character as an input unit, and assign a one-hot representation to it. Other problem settings are the same as previous one. Please see Table 2 for more examples.
In baseline, We use two feed forward neural networks, each for an output. For the auxiliary network, we also use two feed forward neural networks, and average their outputs. Each feed forward network has three hidden layers. The training and other settings for baseline, proposed approach and ablations are the same as the previous experiment.
The results listed in Table 1 (middle) demonstrate that the proposed method is significantly better than the baseline by around 24% absolute increase. We find removing manifold regularization in ablation study shows slight improvement over the proposed approach. Other ablation experiments have significant reductions. This might be because regularizing manifold is not important when inputs (characters) are discrete.

heading: 5.3 EXPERIMENTS ON COLORED DIGITS

text:
We also explore the capability of the proposed approach to another hand written digit problem with different types of components: digits and colors. We construct the dataset from MNIST (LeCun et al., 1998) by changing the color of digits. We use digit label Y1 as a component (0-9), and color label Y2 as another (0-2). Color label is 0, 1, 2 for red, blue, green, respectively. In training, we use label combinations with: Y1 mod 3 6= Y2. In test, we use the rest of the combinations. Please refer to Figure 3 for examples.
We use two compositional neural networks for each output, respectively. Each network has three hidden layers. For the auxiliary network, we concatenated the hidden representation as input, and use a three layer trans-convolutional neural network. Other settings for the methods and ablation are the same as the overlapping digit experiment.
The results shown in Table 1 (right) demonstrate that the proposed method is significantly better than the baseline by more than 40% absolute increase. It also outperforms the ablations, indicating
that all the modifications are necessary. Among them, auxiliary network contributes the most to the performance improvement.

heading: 6 DISCUSSIONS

text: In this section, we perform visualization and error analysis to better understand the experimental results and the behavior of the proposed approach.

heading: 6.1 DISTRIBUTION VISUALIZATION

text:
We visualize hidden representations of both baseline and the proposed approach for overlapping digit experiments (Figure 6). We use t-SNE (Maaten & Hinton, 2008) to reduce each of two hidden representations to one dimension, and jointly plot them along horizontal and vertical coordinates, respectively. Training samples are blue, and test samples are orange.
Our expectation is a chess board like distribution, similar to the true underlying distribution (Figure 4). Note that though there are 10 labels, the expected results may not be 10× 10 colored blocks, because the labels, along with the representation, may not be in order (e.g. switching label 5, 6 reduces two blocks lines), and the first and the second hidden representations may differ for the same digit.
We find the visualization of the proposed approach (Figure 6b) is closer to the expectation than the baseline one (Figure 6a). The proposed approach has less empty areas, indicating that it can recombine the components in the out-of-distribution setting. This analysis demonstrates that the proposed approach works in the expected way.

heading: 6.2 SAMPLE VISUALIZATION

text: We also hope to visualize concrete samples for the proposed approach. Since we have the auxiliary network with two sub-networks for each hidden component representation, we visualize their outputs, and compare with the ground-truth (Figure 7). The result shows that the original input and hidden components are reasonably recovered for both training and test samples. This means that the proposed approach is able to extract information for each component in training distribution, and transfer the ability in test distribution.

heading: 6.3 TRANSFER ERROR ANALYSIS

text:
We analyze errors to show that the inference optimization process helps addressing the transfer problem. When the model makes mistakes in test, the predicted output may correspond to indistribution or out-of-distribution label pairs. We investigate how frequent errors are associated with the distribution transfer. We define a metric of Transfer Error Rate (TER) to measure this. It is the
number of errors predicted to be in wrong distribution, over all errors. In this setting, the sum of labels is odd in test, and even in training, so TER is the number of errors with even predicted label sum over all errors. If this rate is high (100% is upper bound by definition), it means that most of errors are associated with transfer. If it is around 50%, it means that the errors are balanced in distributions.
TER is 89.5±0.7% for baseline, and 68.3±2.8% for the proposed approach, with 500 test samples. Also, before the inference optimization in the proposed approach, the value is 88.8±2.0%. The results show that the baseline has most of the errors to be transfer error, and the propose approach reduces it significantly. Also, the reduction is during optimization in inference, because the value is close to baseline before optimization. This indicates that the proposed approach is effective to reduce TER, and optimization is an important factor for it. Please refer to Figure 5 for more details.

heading: 7 CONCLUSION

text: In this paper, we discuss our finding that compositionality does not transfer naturally from training to test data distributions. We further propose to address this problem with an auxiliary reconstruction network and a regularized optimization procedure during the inference stage. Experimental results show that the proposed approach has more than 20% absolute increase in various experiments comparing with baselines, and even outperforms human on the overlapping digit recognition task. We hope this work would reshape our thoughts on transferability of compositionality and help to advance compositional generalization and artificial intelligence research.

heading: A EXPERIMENT SETTINGS

text:
A.1 EXPERIMENTS ON OVERLAPPING DIGITS
The data of overlapping digits experiment are generated from MNIST dataset (LeCun et al., 1998), which contains 60,000 training data and 10,000 test data. We use the original training data in training and the original test data in test. In training, we use the combinations with even label sum. In test, we use the combinations with odd label sum (evaluation on test distribution). As a reference, we also use the combinations with even label sum (evaluation on training distribution).
Each input image has a shape of 28× 28× 3 for height, width and depth. Extraction network g(·;φ) has two convolutional neural sub-networks. Each of them has three convolutional layers. The first convolutional layer has 3 × 3 kernel and 1 × 1 strides with 32 depth and ReLU activation. It is followed by a max pooling layer with 2× 2 kernel and 1× 1 strides. The second convolutional layer has 3×3 kernel and 1×1 strides with 64 depth and ReLU activation. Then the representation is flatten and fed to a fully connected neural network with 32 hidden nodes and linear activation. Prediction network f(·; θ) has a single layer fully connected neural network with Softmax activation for each pair of hidden representation and output. Auxiliary network h(·;ψ) has two trans-convolutional neural networks. A hidden representation is converted to 7× 7× 32 representation (width, height, depth) by a fully connected neural network with ReLU activation. The first trans-convolutional layer has 3× 3 kernel and 2× 2 strides with 64 depth, SAME padding and ReLU activation. The second trans-convolutional layer has 3× 3 kernel and 2× 2 strides with 32 depth, SAME padding and ReLU activation. The third trans-convolutional layer has 3× 3 kernel and 1× 1 strides with 1 depth, SAME padding and linear activation.
For the hyper parameters, we use α = 1, β = 0.3, γ = 0.03, η = 1, M = 500. In training optimization, we use Adam optimizer (Kingma & Ba, 2014) for 1,000 steps with batch size 500 and learning rate 0.001. In inference optimization, we also use Adam optimizer for 100 steps with batch size 500 and learning rate 0.3. We find the memory size M does not change the result significantly. We have 69.0±0.6% for M = 100, 68.2±1.0% for 300, 69.4±0.3% for 500 (reported), 69.6±0.5% for 700 and 69.9±0.4% for 900 (Figure 8).
A.2 EXPERIMENTS ON COMPOUND WORDS
In the compound words experiment, we have inputs with different lengths. We first patch an additional character symbol to make all inputs the same length of 18. Then, we convert each character to a onehot representation with size 27. we use three layer fully connected neural networks as sub-networks for extractor. The input is flatten and fed to each sub-network. The first layer has 96 hidden nodes with ReLU activation. The second layer has 64 hidden nodes with ReLU activation. The third layer has 32 hidden nodes with linear activation. Prediction network is a single layer fully connected network with Softmax activation for each pair of hidden representation and output. The auxiliary network also has two three layer fully connected sub-networks. The first layer has 64 hidden nodes
with ReLU activation. The second layer has 96 hidden nodes with ReLU activation. The third layer has 486 hidden nodes with linear activation. Then the nodes are reshaped to 18× 27. For the hyper parameters, we use α = 1, β = 0.1, γ = 0.01, η = 1, M = 500. In training optimization, we use Adam optimizer for 1,000 steps with batch size 500 and learning rate 0.001. In inference optimization, we also use Adam optimizer for 100 steps with batch size 500 and learning rate 0.3.
A.3 EXPERIMENTS ON COLORED DIGITS
The data for colored digits experiment are created from MNIST dataset. Each input image is in shape of 28 × 28 × 3 for height, width and depth. The extractor has two sub-networks, each has three convolutional layers. The first convolutional layer has 3× 3 kernel and 1× 1 strides with 32 depth and ReLU activation. A max pooling layer follows it with 2× 2 kernel and 1× 1 strides. The second convolutional layer has 3× 3 kernel and 1× 1 strides with 64 depth and ReLU activation. Then the representation is flatten and fed to a fully connected neural network with 32 hidden nodes with linear activation. Prediction network has a fully connected network for each pair of hidden representation and output with Softmax activation. For auxiliary network, we first concatenate the hidden representations to have 64 nodes. It is then converted to 7 × 7 × 32 representation (width, height, depth) by a fully connected neural network with ReLU activation. The first trans-convolutional layer has 3× 3 kernel and 2× 2 strides with 64 depth, SAME padding and ReLU activation. The second trans-convolutional layer has 3× 3 kernel and 2× 2 strides with 32 depth, SAME padding and ReLU activation. The third trans-convolutional layer has 3× 3 kernel and 1× 1 strides with 3 depth, SAME padding and linear activation.
For the hyper parameters, we use α = 1, β = 0.3, γ = 0.03, η = 1, M = 500. In training optimization, we use Adam optimizer for 1,000 steps with batch size 500 and learning rate 0.001. In inference optimization, we also use Adam optimizer for 100 steps with batch size 500 and learning rate 0.3.

heading: B CROWD SOURCING FOR HUMAN PERFORMANCE

text: We collect data on human performance through crowd-sourced survey. The survey has training and test sections. We select 20 images uniformly at random from the training distribution as training data. We also select 20 images uniformly at random from the test distribution as test data (Figure 9). In the training section we state that each picture is an overlap of two pictures containing two numbers. Then we ask survey takers to discover the rule through the training examples before they answer questions. In the test section we put up test images, and two multiple-selection (0-9) list boxes under each image. Then we ask the survey takers to identify the correct answers. In this survey we had 27 responses. The mean test accuracy is 10.0% with standard deviation of 7.7%.