HEALTHCARE REPRESENTATION LEARNING
Deep learning methods exhibit promising performance for predictive modeling in healthcare, but two important challenges remain: • Data insufficiency: Often in healthcare predictive modeling, the sample size is insufficient for deep learning methods to achieve satisfactory results. • Interpretation: The representations learned by deep learning models should align with medical knowledge. To address these challenges, we propose a GRaph-based Attention Model, GRAM that supplements electronic health records (EHR) with hierarchical information inherent to medical ontologies. Based on the data volume and the ontology structure, GRAM represents a medical concept as a combination of its ancestors in the ontology via an attention mechanism. We compared predictive performance (i.e. accuracy, data needs, interpretability) of GRAM to various methods including the recurrent neural network (RNN) in two sequential diagnoses prediction tasks and one heart failure prediction task. Compared to the basic RNN, GRAM achieved 10% higher accuracy for predicting diseases rarely observed in the training data and 3% improved area under the ROC curve for predicting heart failure using an order of magnitude less training data. Additionally, unlike other methods, the medical concept representations learned by GRAM are well aligned with the medical ontology. Finally, GRAM exhibits intuitive attention behaviors by adaptively generalizing to higher level concepts when facing data insufficiency at the lower level concepts.
insufficient for deep learning methods to achieve satisfactory results. • Interpretation: The representations learned by deep learning models should
align with medical knowledge. To address these challenges, we propose a GRaph-based Attention Model, GRAM that supplements electronic health records (EHR) with hierarchical information inherent to medical ontologies. Based on the data volume and the ontology structure, GRAM represents a medical concept as a combination of its ancestors in the ontology via an attention mechanism. We compared predictive performance (i.e. accuracy, data needs, interpretability) of GRAM to various methods including the recurrent neural network (RNN) in two sequential diagnoses prediction tasks and one heart failure prediction task. Compared to the basic RNN, GRAM achieved 10% higher accuracy for predicting diseases rarely observed in the training data and 3% improved area under the ROC curve for predicting heart failure using an order of magnitude less training data. Additionally, unlike other methods, the medical concept representations learned by GRAM are well aligned with the medical ontology. Finally, GRAM exhibits intuitive attention behaviors by adaptively generalizing to higher level concepts when facing data insufficiency at the lower level concepts.
1 INTRODUCTION
The rapid growth in volume and diversity of health care data from electronic health records (EHR) and other sources is motivating the use of predictive modeling to improve care for individual patients. In particular, novel applications are emerging that use deep learning methods such as word embedding (Choi et al., 2016c;e), recurrent neural networks (RNN) (Che et al., 2016; Choi et al., 2016a;b; Lipton et al., 2016), convolutional neural networks (CNN) (Nguyen et al., 2016) or stacked denoising autoencoders (SDA) (Che et al., 2015; Miotto et al., 2016), demonstrating significant performance enhancement for diverse prediction tasks. Deep learning models appear to perform significantly better than logistic regression or multilayer perceptron (MLP) models that depend, to some degree, on expert feature construction (Lipton et al., 2015; Razavian et al., 2016).
Training deep learning models typically requires large amounts of data that often cannot be met by a single health system or provider organization. Sub-optimal model performance can be particularly challenging when the focus of interest is predicting onset of a specific disease (e.g. heart failure) or related events such as accelerated disease progression. For example, using Doctor AI (Choi et al., 2016a), we discovered that RNN alone was ineffective to predict the onset of diseases such as cerebral degenerations (e.g. Leukodystrophy, Cerebral lipidoses) or developmental disorders (e.g. autistic disorder, Heller’s syndrome), partly because their rare occurrence in the training data provided little learning opportunity to the flexible models like RNN.
The data requirement of deep learning models comes from having to assess exponential number of combinations of input features. This can be alleviated by exploiting medical ontologies that encodes hierarchical clinical constructs and relationships among medical concepts. Fortunately, there are many well-organized ontologies in healthcare such as the International Classification of Diseases (ICD), Clinical Classifications Software (CCS) (Stearns et al., 2001) or Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT) (Project et al., 2010). Nodes (i.e. medical concepts) close to one another in medical ontologies are likely to be associated with similar patients, allowing
us to transfer knowledge among them. Therefore, proper use of medical ontologies will be helpful when we lack enough data for the nodes in the ontology to train deep learning models.
In this work, we propose GRAM, a method that infuses information from medical ontologies into deep learning models via neural attention. Considering the frequency of a medical concept in the EHR data and its ancestors in the ontology, GRAM decides the representation of the medical concept by adaptively combining its ancestors via attention mechanism. This will not only support deep learning models to learn robust representations without large amount of data, but also learn interpretable representations that align well with the knowledge from the ontology. The attention mechanism is trained in an end-to-end fashion with the neural network model that predicts the onset of disease(s). We also propose an effective initialization technique in addition to the ontological knowledge to better guide the representation learning process.
We compared predictive performance (i.e. accuracy, data needs, interpretability) of GRAM to various models including the recurrent neural network (RNN) in two sequential diagnoses prediction tasks and one heart failure (HF) prediction task. We demonstrate that GRAM is up to 10% more accurate than the basic RNN for predicting diseases less observed in the training data. After discussing GRAM’s scalability, we visualize the representations learned from various models where GRAM provides more intuitive representations by grouping similar medical concepts close to one another. Finally, we show GRAM’s attention mechanism can be interpreted to understand how it assigns the right amount of attention to the ancestors of each medical concept by considering the data availability and the ontology structure.
2 METHODOLOGY
We first define the notations describing EHR data and medical ontologies, followed by a description of GRAM (Section 2.2), the end-to-end training of the attention generation and predictive modeling (Section 2.3), and the efficient initialization scheme (Section 2.4).
2.1 BASIC NOTATION
We denote the set of entire medical codes from the EHR as c1, c2, . . . , c|C| ∈ C with the vocabulary size |C|. The clinical record of each patient can be viewed as a sequence of visits V1, . . . , VT where each visit contains a subset of medical codes Vt ⊆ C. Vt can be represented as a binary vector xt ∈ {0, 1}|C| where the i-th element is 1 only if Vt contains the code ci. To avoid clutter, all algorithms will be presented for a single patient.
We assume that a given medical ontology G typically expresses the hierarchy of various medical concepts in the form of a parent-child relationship, where the medical codes C form the leaf nodes. Ontology G is represented as a directed acyclic graph (DAG) whose nodes form a set D = C + C′. C′ = {c|C|+1, c|C|+2, . . . , c|C|+|C′|} defines the set of all non-leaf nodes (i.e. ancestors of the leaf nodes), where |C′| represents the number of all non-leaf nodes. We use knowledge DAG to refer to G. A parent in the knowledge DAG G represents a related but more general concept over its children. Therefore, G provides a multi-resolution view of medical concepts with different degrees of specificity. While some ontologies are exclusively expressed as parent-child hierarchies (e.g. ICD-9, CCS), others are not. For example, in some instances SNOMED-CT also links medical concepts to causal or treatment relationships, but the majority relationships in SNOMED-CT are still parent-child. Therefore, we focus on the parent-child relationships in this work.
2.2 KNOWLEDGE DAG AND THE ATTENTION MECHANISM
GRAM leverages the parent-child relationship of G to learn robust representations when data volume is constrained. GRAM balances the use of ontology information in relation to data volume in determining the level of specificity for a medical concept. When a medical concept is less observed in the data, more weight is given to its ancestors as they can be learned more accurately and offer general (coarse-grained) information about their children. The process of resorting to the parent concepts can be automated via the attention mechanism and the end-to-end training as described in Figure 1.
In the knowledge DAG, each node ci is assigned a basic embedding vector ei ∈ Rm, where m represents the dimensionality. Then e1, . . . , e|C| are the basic embeddings of the codes c1, . . . , c|C| while e|C|+1, . . . , e|C|+|C′| represent the basic embeddings of the internal nodes c|C|+1, . . . , c|C|+|C′|. The initialization of these basic embeddings is described in Section 2.4. We formulate a leaf node’s
final representation as a convex combination of the basic embeddings of itself and its ancestors: gi = ∑
j∈A(i)
αijej , ∑
j∈A(i)
αij = 1, αij ≥ 0 for j ∈ A(i), (1)
where gi ∈ Rm denotes the final representation of the code ci,A(i) the indices of the code ci and ci’s ancestors, ej the basic embedding of the code cj and αij ∈ R the attention weight on the embedding ej when calculating gi. The attention weight αij in Eq. (1) is calculated by the following Softmax function,
αij = exp(f(ei, ej))∑
k∈A(i) exp(f(ei, ek)) (2)
f(ei, ej) is a scalar value representing the compatibility between the basic embeddings of ei and ek. We compute f(ei, ej) via the following feed-forward network with a single hidden layer (MLP),
f(ei, ej) = u > a tanh(Wa [ ei ej ] + ba) (3)
where Wa ∈ Rl×2m is the weight matrix for the concatenation of ei and ej , b ∈ Rl the bias vector, and ua ∈ Rl the weight vector for generating the scalar value. The constant l represents the dimension size of the hidden layer of f(·, ·). Note that we always concatenate ei and ej in the child-ancestor order.
Remarks: The example in Figure 1 is derived based on a single path from ci to ca. However, the same mechanism can be applicable to multiple paths as well. For example, code ck has two paths to the root ca, containing five ancestors in total. Another scenario is where the EHR data contain both leaf codes and some ancestor codes. We can move those ancestors present in EHR data from the set C′ to C and apply the same process as Eq. (1) to obtain the final representations for them.
2.3 END-TO-END TRAINING WITH A PREDICTIVE MODEL
We train the attention mechanism together with a predictive model such that the attention mechanism improves the predictive performance. Once the final representations g1,g2, . . . ,g|C| of all medical codes are obtained, we can convert visit Vt to a visit representation vt by using the embedding matrix G ∈ Rm×|C| where gi is its i-th column as in Figure 1. We continue the mathematical formulation under the assumption that we are using the RNN to perform sequential diagnoses prediction (Choi et al., 2016a;b) with the objective of predicting the disease codes of the next visit Vt+1 given the visit records up to the current timestep V1, V2, . . . , Vt, which can be expressed as follows,
ŷt = x̂t+1 = Softmax(Wht + b), where h1,h2, . . . ,ht = RNN(v1,v2, . . . ,vt), where (4) v1,v2, . . . ,vt = tanh(G[x1,x2, . . . ,xt])
where xt ∈ R|C| denotes the t-th visit; vt ∈ Rm the t-th visit representation; ht ∈ Rr the RNN’s hidden layer at t-th time step (i.e. t-th visit); W ∈ R|C|×r and b ∈ R|C| the weight matrices and the bias vector of the Softmax function; r denotes the dimension size of the hidden layer. We use “RNN” to denote any recurrent neural network variants that can cope with the vanishing gradient problem (Bengio et al., 1994), such as LSTM (Hochreiter & Schmidhuber, 1997), GRU (Cho et al., 2014), and IRNN (Le et al., 2015), with any varying numbers of hidden layers. The prediction loss for all time steps is calculated using the cross entropy as follows, L(x1,x2 . . . ,xT ) = − 1T−1 ∑T−1 t=1 ( yt > log(ŷt) + (1− yt)> log(1− ŷt) ) where we sum the cross entropy errors from
all dimensions of ŷt, T denotes the length of the visit sequence. Note that the above loss is defined for a single patient. But we can take the average of the individual loss for multiple patients.
2.4 INITIALIZING BASIC EMBEDDINGS
The attention generation mechanism in Section 2.2 requires basic embeddings ei of each node in the knowledge DAG. The basic embeddings of ancestors, however, pose a difficulty because they are often not observed in the data.To better initialize them, we use co-occurrence information to learn the basic embeddings of medical codes and their ancestors. Co-occurrence has proven to be an important source of information when learning representations of words or medical concepts (Mikolov et al., 2013; Choi et al., 2016c;e). To train the basic embeddings, we employ GloVe (Pennington et al., 2014), which uses the global co-occurrence matrix of words to learn their representations. In our case, the co-occurrence matrix of the codes and the ancestors was generated by counting the co-occurrences within each visit Vt, where we augment each visit with the ancestors of the codes in the visit. Details of training the basic embeddings are described in the Appendix A. Note that, with or without the initialization, the basic embeddings ei’s of both leaf nodes (i.e. medical codes) and non-leaf nodes (i.e. ancestors) are fine-tuned when training our model, since the error signal flows from the output ŷt to the final representations gi’s which are convex combinations of ei’s.
3 EXPERIMENTS
We conduct three experiments to determine if GRAM offered superior prediction performance when facing data insufficiency. We first describe the experimental setup followed by results comparing predictive performance of GRAM with various baseline models. After discussing GRAM’s scalability, we qualitatively evaluate the interpretability of the resulting representation. The source code of GRAM is publicly available at https://github.com/mp2893/gram.
3.1 EXPERIMENT SETUP
Prediction tasks and source of data: We conduct two sequential diagnoses prediction tasks, which aim at predicting all diagnosis categories in the next visit, and one heart failure (HF) prediction task, which is a binary prediction task for predicting a future HF onset where the prediction is made only once at the last visit xT . Two sequential diagnoses predictions are respectively conducted using 1) Sutter Palo Alto Medical Foundation (PAMF) dataset, which consists of 18-years longitudinal medical records of 258K patients between age 50 and 90. This will determine GRAM’s performance for general adult population with long visit records. 2) MIMIC-III dataset (Johnson et al., 2016; Goldberger et al., 2000), which is a publicly available dataset consisting of medical records of 7.5K intensive care unit (ICU) patients over 11 years. This will determine GRAM’s performance for high-risk patients with very short visit records. We utilize all the patients with at least 2 visits. We prepared the true labels yt by grouping the ICD9 codes into 283 groups using CCS single-level diagnosis grouper1. This is to improve the
1https://www.hcup-us.ahrq.gov/toolssoftware/ccs/AppendixASingleDX.txt
training speed and predictive performance for easier analysis, while preserving sufficient granularity for each diagnosis. Each diagnosis code’s varying frequency in the training data can be viewed as different degrees of data insufficiency. We calculate Accuracy@k for each of CCS single-level diagnosis codes such that, given a visit Vt, we get 1 if the target diagnosis is in the top k guesses and 0 otherwise. We conduct HF prediction on Sutter heart failure (HF) cohort, which is a subset of Sutter PAMF data for a heart failure onset prediction study with 3.4K HF cases and 27K controls chosen by a set of criteria (see Appendix B). This will determine GRAM’s performance for a different prediction task where we predict the onset of one specific condition. We randomly downsample the training data to create different degrees of data insufficiency. We use area under the ROC curve (AUC) to measure the performance. A summary of the datasets are provided in Table 1.We used CCS multi-level diagnoses hierarchy2 as our knowledge DAG G. We also tested the ICD9 code hierarchy3, but the performance was similar to using CCS multi-level hierarchy. For all three tasks, we randomly divide the dataset into the training, validation and test set by .75:.10:.15 ratio, and use the validation set to tune the hyper-parameters. Further details regarding the hyper-parameter tuning are provided in Appendix C. The test set performance is reported in the paper.
Implementation details: We implemented GRAM with Theano 0.8.2 (Team, 2016). For training models, we used Adadelta (Zeiler, 2012) with a mini-batch of 100 patients, on a machine equipped with Intel Xeon E5-2640, 256GB RAM, four Nvidia Titan X’s and CUDA 7.5.
Models for comparison are the following. The first two GRAM+ and GRAM are the proposed methods and the rest are baselines. Hyper-parameter tuning is configured so that the number of parameters for the baselines would be comparable to GRAM’s. Further details are provided in Appendix C.
• GRAM: Input sequence x1, . . . ,xT is first transformed by the embedding matrix G, then fed to the GRU with a single hidden layer, which in turn makes the prediction, as described by Eq. (4). The basic embeddings ei’s are randomly initialized.
• GRAM+: We use the same setup as GRAM, but the basic embeddings ei’s are initialized according to Section 2.4.
• RandomDAG: We use the same setup as GRAM, but each leaf concept has five randomly assigned ancestors from the CCS multi-level hierarchy to test the effect of correct domain knowledge.
• RNN: Input xt is transformed by an embedding matrix Wemb ∈ Rk×|C|, then fed to the GRU with a single hidden layer. The embedding size k is a hyper-parameter. Wemb is randomly initialized and trained together with the GRU.
• RNN+: We use the same setup as RNN, but we initialize the embedding matrix Wemb with GloVe vectors trained only with the co-occurrence of leaf concepts. This is to compare GRAM with a similar weight initialization technique. • SimpleRollUp: We use the same setup as RNN. But for input xt, we replace all diagnosis codes
with their direct parent codes in the CCS multi-level hierarchy, giving us 578, 526 and 517 input codes respectively for Sutter data, MIMIC-III and Sutter HF cohort. This is to compare the performance of GRAM with a common grouping technique.
• RollUpRare: We use the same setup as RNN, but we replace any diagnosis code whose frequency is less than a certain threshold in the dataset with its direct parent. We set the threshold to 100 for Sutter data and Sutter HF cohort, and 10 for MIMIC-III, giving us 4,408, 935 and 1,538 input codes respectively for Sutter data, MIMIC-III and Sutter HF cohort. This is an intuitive way of dealing with infrequent medical codes.
3.2 PREDICTION PERFORMANCE AND SCALABILITY
Tables 2a and 2b show the sequential diagnoses prediction performance on Sutter data and MIMIC-III. Both figures show that GRAM+ outperforms other models when predicting labels with significant data insufficiency (i.e. less observed in the training data).The performance gain is greater for MIMIC-III, where GRAM+ outperforms the basic RNN by 10% in the 20th-40th percentile range. This seems to come from the fact that MIMIC patients on average have significantly shorter visit history than Sutter patients, with much more codes received per visit. Such short sequences make it difficult for the RNN to learn and predict diagnoses sequence. The performance difference between GRAM+ and
2https://www.hcup-us.ahrq.gov/toolssoftware/ccs/AppendixCMultiDX.txt 3http://www.icd9data.com/2015/Volume1/default.htm
GRAM suggests that our proposed initialization scheme of the basic embeddings ei is important for sequential diagnosis prediction.
Table 2c shows the HF prediction performance on Sutter HF cohort. GRAM and GRAM+ consistently outperforms other baselines (except RNN+) by 3∼4% AUC, and RNN+ by maximum 1.8% AUC. These differences are quite significant given that the AUC is already in the mid-80s, a high value for HF prediction, cf. (Choi et al., 2016d). Note that, for GRAM+ and RNN+, we used the downsampled training data to initialize the basic embeddings ei’s and the embedding matrix Wemb with GloVe, respectively. The result shows that the initialization scheme of the basic embeddings in GRAM+ gives limited improvement over GRAM. This stems from the different natures of the two prediction tasks. While the goal of HF prediction is to predict a binary label for the entire visit sequence, the goal of sequential diagnosis prediction is to predict the co-occurring diagnosis codes at every visit. Therefore the co-occurrence information infused by the initialized embedding scheme is more beneficial to sequential diagnosis prediction. Additionally, this benefit is associated with the natures of the two prediction tasks than the datasets used for the prediction tasks. Because the initialized embedding shows different degrees of improvement as shown by Tables 2a and 2c, when Sutter HF cohort is a subset of Sutter PAMF, thus having similar characteristics. Additional prediction results when varying the k of Accuracy@k are discussed in the Appendix D.
Overall, GRAM showed superior predictive performance under data insufficiency in three different experiments, demonstrating its general applicability in predictive healthcare modeling. Now we briefly discuss the scalability of GRAM by comparing its training time to RNN’s. Table 3 shows the number of seconds taken for the two models to train for a single epoch for each predictive modeling task. GRAM+ and RNN+ showed the same behavior as GRAM and RNN. GRAM takes approximately 50% more time to train for a single epoch for all prediction tasks. This stems from calculating attention weights and the final representations gi for all medical codes. GRAM also generally takes about 50% more epochs to reach to the model with the lowest validation loss. This is due to optimizing an extra MLP model that generates the attention weights. Overall, use of GRAM adds a manageable amount of overhead in training time to the plain RNN.
3.3 QUALITATIVE EVALUATION OF INTERPRETABLE REPRESENTATIONS
To qualitatively assess the interpretability of the learned representations of the medical codes, we plot on a 2-D space using t-SNE (Maaten & Hinton, 2008) the final representations gi of 2,000
randomly chosen diseases learned by GRAM+ for sequential diagnoses prediction on Sutter data4 (Figure 2a). The colors represent the highest disease categories and the text annotations represent the detailed disease categories in CCS multi-level hierarchy. For comparison, we also show the t-SNE plots on the strongest results from RNN+ (Figure 2b), and GloVe (Figure 2c), the same embedding technique in initializing the basic embeddings ei. Figures 2b and 2c confirm that interpretable representations cannot simply be learned only by co-occurrence or supervised prediction without medical knowledge. GRAM+ learns disease representations that are significantly more consistent with the given knowledge DAG G. Therefore the neural network predictive model that accepts gi is using accurate representations that lead to higher predictive performance. Additional scatterplots of other models are provided in Appendix E for comparison. An interactive visualization tool can be accessed at http://www.sunlab.org/research/gram-graph-based-attention-model/.
3.4 ANALYSIS OF THE ATTENTION BEHAVIOR
Next we show that GRAM’s attention can be interpreted to understand how it considers data availability and knowledge DAG’s structure when performing a prediction task. Using Eq. (1), we can calculate the attention weights of individual disease. Figure 3 shows the attention behaviors of four representative diseases when performing HF prediction on Sutter HF cohort.
Other pneumothorax (ICD9 512.89) in Figure 3a is rarely observed in the data and has only five siblings. In this case, most information is derived from the highest ancestor. Temporomandibular joint disorders & articular disc disorder (ICD9 524.63) in Figure 3b is rarely observed but has 139 siblings. In this case, its parent receives a stronger attention because it aggregates sufficient samples from all of its children to learn a more accurate representation. Note that the disease itself also receives a stronger attention to facilitate easier distinction from its large number of siblings.
4The scatterplots of models trained for sequential diagnoses prediction on MIMIC-III and HF prediction for Sutter HF cohort were similar but less structured due to smaller data size.
Unspecified essential hypertension (ICD9 401.9) in Figure 3c is very frequently observed but has only two siblings. In this case, GRAM assigns a very strong attention to the leaf, which is logical because the more you observe a disease, the stronger your confidence becomes. Need for prophylactic vaccination and inoculation against influenza (ICD9 V04.81) in Figure 3d is quite frequently observed and also has 103 siblings. The attention behavior in this case is quite similar to the case with fewer siblings (Figure 3b) with a slight attention shift towards the leaf concept as more observations lead to higher confidence.
4 RELATED WORK
We introduce recent studies related to GRAM that learn the representations of graphs and discuss their relationship with GRAM. Several studies focused on learning the representations of graph vertices by using the neighbor information. DeepWalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016) use random walk while LINE (Tang et al., 2015) uses breadth-first search to find the neighbors of a vertex and learn its representation based on the neighbor information. Graph convolutional approaches (Yang et al., 2016; Kipf & Welling, 2016) also focus on learning the vertex representations to mainly perform vertex classification. These works focus on solving the graph data problems whereas GRAM focuses on solving EHR data problems using the knowledge DAG as supplementary information.
Several researchers tried to model the knowledge DAG such as WordNet (Miller, 1995) or Freebase (Bollacker et al., 2008) where two entities are connected with various types of relation, forming a set of triples. They aim to project entities and relations (Bordes et al., 2013; Socher et al., 2013; Wang et al., 2014; Lin et al., 2015) to the latent space based on the triples or additional information such as hierarchy of entities (Xie et al., 2016). These works demonstrated tasks such as link prediction, triple classification or entity classification using the learned representations. More recently, Li et al. (2016) learned the representations of words and Wikipedia categories by utilizing the hierarchy of Wikipedia categories. GRAM is fundamentally different from the above studies in that it aims to design intuitive attention mechanism on the knowledge DAG as a knowledge prior to cope with data insufficiency and learn medically interpretable representations to make accurate predictions.
A classical approach for incorporating side information in the predictive models is to use graph Laplacian regularization (Weinberger et al., 2006; Che et al., 2015). However, using this approach is not straightforward as it relies on the appropriate definition of distance on graphs which is often unavailable.
5 CONCLUSION
Data insufficiency, either due to less common diseases or small datasets, is one of the key hurdles in healthcare analytics, especially when we apply deep neural networks models. To overcome this hurdle, we leverage the knowledge DAG, which provides a multi-resolution view of medical concepts. We propose GRAM, a graph-based attention model using both a knowledge DAG and EHR to learn an accurate and interpretable representations for medical concepts. GRAM chooses a weighted average of ancestors of a medical concept and train the entire process with a predictive model in an end-to-end fashion. We conducted three predictive modeling experiments on real EHR datasets and showed significant improvement in the prediction performance, especially on low-frequency diseases and small datasets. Analysis of the attention behavior provided intuitive insight of GRAM.
A GENERATING GLOVE EMBEDDINGS
We learn the basic embeddings ei’s of medical codes and their ancestors using GloVe (Pennington et al., 2014), which uses global co-occurrence matrix of words to learn their representations. We generate the co-occurrence matrix of the codes and the ancestors by counting the co-occurrence within each visit Vt. However, since visits only contain the leaf codes c ∈ C, we augment each visit with the ancestors of the codes in each visit, then count the co-occurrence of codes and ancestors altogether.
We describe the details of the algorithm with an example. We borrow the parent-child relationships from the knowledge DAG of Figure 1. Given a visit Vt,
Vt = {cd, ci, ck} (5) we augment it with the ancestors of all the codes to obtain the augmented visit V ′t ,
V ′t = {cd, cb, ca, ci, cg, cc, ca, ck, cj , cf , cc, cb, ca} (6)
where the added ancestors are underlined. Note that a single ancestor can appear multiple times in V ′t . In fact, the higher the ancestor is in the knowledge DAG, the more times it is likely to appear in V ′t . We count the co-occurrence of two codes in V ′ t as follows,
co-occurrence(ci, cj , V ′t ) = count(ci, V ′ t )× count(cj , V ′t ) (7)
where count(ci, V ′t ) is the number of times the code ci appears in the augmented visit V ′ t . For example, the co-occurrence between the leaf code ci and the root ca is 3. However, the co-occurrence between the ancestor cc and the root ca is 6. Therefore our algorithm will naturally make the ancestor codes have higher co-occurrence with other codes compared to leaf medical codes. We repeat this calculation for all pairs of codes in all augmented visits of all patients to obtain the co-occurrence matrix depicted by Figure 4. For training the embedding vectors using the co-occurrence matrix, we use the same procedure and hyper-parameter as described in Pennington et al. (2014).
B HEART FAILURE COHORT CONSTRUCTION
For the heart failure (HF) case patients, we select patients between 40 to 85 years of age at the time of HF diagnosis. HF diagnosis (HFDx) criteria are defined as: 1) Qualifying ICD-9 codes for HF appeared in the encounter records or medication orders. Qualifying ICD-9 codes are listed in Table 4. 2) at least three clinical encounters with qualifying ICD-9 codes had to occur within 12 months of each other, where the date of HFDx was assigned to the earliest of the three dates. If the time span between the first and second appearances of the HF diagnosis code was greater than 12 months, the date of the second encounter was used as the first qualifying encounter. Up to ten eligible controls (in terms of sex, age, location) were selected for each case, yielding average 9 controls per case. Each control was also assigned an index date, which is the HFDx date of the matched case. Controls are selected such that they did not meet the HF diagnosis criteria prior to the HFDx date plus 182 days of
their corresponding case. Control subjects were required to have their first office encounter within one year of the matching HF case patient’s first office visit, and have at least one office encounter 30 days before or any time after the case’s HFDx date to ensure similar duration of observations among cases and controls.
C HYPER-PARAMETER TUNING
We define five hyper-parameters for GRAM:
• dimensionality m of the basic embedding ei: [100, 200, 300, 400, 500] • dimensionality r of the RNN hidden layer ht from Eq. (4): [100, 200, 300, 400, 500] • dimensionality l of Wa and ba from Eq. (3): [100, 200, 300, 400, 500] • L2 regularization coefficient for all weights except RNN weights: [0.1, 0.01, 0.001, 0.0001] • dropout rate for the dropout on the RNN hidden layer: [0.0, 0.2, 0.4, 0.6, 0.8]
We performed 100 iterations of the random search by using the above ranges for each of the three prediction experiments. For sequential diagnoses prediction on Sutter data, we used 10% of the training data to tune the hyper-parameters to balance the time and search space. To match the baselines’ number of parameters to GRAM’s, we add 550 to the list of m’s possible values. This will make the baseline’s largest possible number of parameters comparable to the GRAM’s largest possible number of parameters.
For SimpleRollUp and RollUpRare, the number of input codes is smaller than other models due to the grouping. Therefore, to match their largest possible number of parameters to GRAM’s, we need to add much larger values to m. However, after preliminary experiments, as expected, setting m to too
large a value degraded the performance due to overfitting. Since the number of input codes decreased due to the grouping, increasing the dimensionality of ei is not a logical thing to do. Therefore, for SimpleRollUp and RollUpRare, we use the same list of values for m as other baselines.
Table 5 describes the final hyper-parameter settings we used for all models for each prediction experiments.
D PREDICTION RESULTS USING DIFFERENT k’S IN ACCURACY@K
We show Accuracy@k using k = 5, 10, 20, 30 for sequential diagnoses prediction on Sutter data (Tables 6a, 6b, 6c and 6d) and MIMIC-III (Tables 7a, 7b, 7c and 7d). We can see from the tables that GRAM+ consistently outperforms other models under 40th percentile range, except when k = 20, 30 for sequential diagnoses prediction on Sutter data where SimpleRollUp shows similar performance. We can also see that GRAM+ performs significantly better than other models for all k = 5, 10, 20, 30 when predicting infrequently observed diseases on MIMIC-III. As discussed in Section 3.2, this seems to come from the short visit sequences of MIMIC patients.
E T-SNE 2-D PLOTS OF VARIOUS MODELS
For further comparison, we display t-SNE scatterplots of GRAM (Figure 5a) RandomDAG (Figure 5b, RNN (Figure 5c), and Skip-gram (Figure 5d). GRAM, RandomDAG and RNN were trained for sequential diagnoses prediction on Sutter data, and Skip-gram (Mikolov et al., 2013) was trained on Sutter data as it is an unsupervised method. For Skip-gram, we used each visit Vt as the context window. As we do not distinguish between the target concept and the neighbor concepts, we calculated the Skip-gram objective function using all possible pairs of codes within a single visit.
We can see from Figure 5a that the quality of the final representations gi of GRAM is quite similar to GRAM+ (Figure 2a). Compared to other baselines, GRAM demonstrates significantly more structured representations that align well with the given knowledge DAG. It is interesting that Skip-gram shows the most structured representation among all baselines. We used GloVe to initialize the basic embeddings ei in this work because it uses global co-occurrence information and its training time is dependent only on the total number of unique concepts |C|. Skip-gram’s training time, on the other hand, depends on both the number of patients and the number of visits each patient made, which makes the algorithm generally slower than GloVe. However, considering both Figures 2c and 5d, initializing ei’s with Skip-gram vectors might give us additional performance boost.

SUMMARY.
This paper presents a method for enriching medical concepts with their parent nodes in an ontology.
The method employs an attention mechanism over the parent nodes of a medical concept to create a richer representation of the concept itself.
The rationale of this is that for  infrequent medical concepts the attention mechanism will rely more on general concepts, higher in the ontology hierarchy, while for frequent ones will focus on the specific concept.
The attention mechanism is trained together with a recurrent neural network and the model accuracy is tested on two tasks.
The first task aims at prediction the diagnosis categories at each time step, while the second task aims at predicting whether or not a heart failure is likely to happen after the T-th step.
Results shows that the proposed model works well in condition of data insufficiency.
----------
OVERALL JUDGMENT
The proposed model is simple but interesting.
The ideas presented are worth to expand but there are also some points where the authors could have done better.
The learning of the representation of concepts in the ontology is a bit naive, for example the authors could have used some kind of knowledge base factorization approach to learn the concepts, or some graph convolutional approach.
I do not see why the the very general factorization methods for knowledge bases do not apply in the case of ontology learning.
I also found strange that the representation of leaves are fine tuned while the inner nodes are not, it is a specific reason to do so?
Regarding the presentation, the paper is clear and the qualitative evaluation is insightful.
----------
DETAILED COMMENTS
Figure 2. Please use the same image format with the same resolution.


We develop a model of perceptual similarity judgment based on re-training a deep convolution neural network (DCNN) that learns to associate different views of each 3D object to capture the notion of object persistence and continuity in our visual experience. The re-training process effectively performs distance metric learning under the object persistency constraints, to modify the view-manifold of object representations. It reduces the effective distance between the representations of different views of the same object without compromising the distance between those of the views of different objects, resulting in the untangling of the view-manifolds between individual objects within the same category and across categories. This untangling enables the model to discriminate and recognize objects within the same category, independent of viewpoints. We found that this ability is not limited to the trained objects, but transfers to novel objects in both trained and untrained categories, as well as to a variety of completely novel artificial synthetic objects. This transfer in learning suggests the modification of distance metrics in viewmanifolds is more general and abstract, likely at the levels of parts, and independent of the specific objects or categories experienced during training. Interestingly, the resulting transformation of feature representation in the deep networks is found to significantly better match human perceptual similarity judgment than AlexNet, suggesting that object persistence potentially could be a important constraint in the development of perceptual similarity judgment in our brains.
1 INTRODUCTION
The power of the human mind in inference and generalization rests on our brain’s ability to develop models of abstract knowledge of the natural world (Tenenbaum et al., 2011). When shown novel objects, both children and adults can rapidly generalize from just a few examples to classify and group them based on their perceptual similarity. Understanding the processes that give rise to perceptual similarity will provide insight into the development of abstract models in our brain. In this paper, we explored computational models for understanding the neural basis of human perceptual similarity judgment.
Recent deep convolutional neural networks (DCNNs) have produced feature representations in the hidden layers that can match well with neural representations observed in the primate and human visual cortex. It was found that there is a strong correspondence between neural activities (neuronal spikes or fMRI signals) and the activities of the deep layers of deep networks (Agrawal et al., 2014; Khaligh-Razavi & Kriegeskorte, 2014; Yamins et al., 2014), suggesting that deep neural networks have in fact learned meaningful representations that are close to humans’, even though the neural
networks are trained for object classification in computer vision. Cognitive neuroscientists have started to explore how the representations learned by deep networks can be used to model various aspects of human perception such as memorability of objects in images (Dubey et al., 2015), object typicality (Lake et al., 2015), and similarity judgment (Peterson et al., 2016; Kubilius et al., 2016). Certain correspondence between deep net representations and human experimental results are found. In particular, Peterson et al. (2016) found that human similarity judgment on a set of natural images might be similar to the feature representations in deep networks after some transformation.
The DCNNs that neuroscientists and cognitive scientists have studied so far, such as AlexNet (Krizhevsky et al., 2012), were trained with static images with the goal of classifying objects in static images into different categories. Perceptual similarity judgment is obviously closely related to the mechanisms used in object classification—we classify objects with similar attributes and appearances into the same class, and thus object classification rests in part on our perceptual similarity judgment and relies on physical, semantic abstract attributes common to objects in each class. Our perceptual similarity judgment might also be tied to our need for individual object recognition—after all, we might want to recognize an individual person or object, not just a class. It is obviously important to be able to recognize one’s own child or the cup one is using. The need to recognize an individual object, independent of view points, requires fine discrimination of details, and might also be a very potent force for shaping our perceptual similarity judgment’s machinery.
The development of invariant object recognition has often been attributed to object continuity or persistence in our visual experience. When we see an object, we tend to see it from different angles over time, as we walk by or around it, or directly manipulate it. This temporal persistence of objects allows our visual system to associate one view of an object with another view of the same object experienced in temporal proximity, as were proposed in slow-feature analysis (Wiskott & Sejnowski, 2002) or memory trace models (Perry et al., 2006) in computational neuroscience for learning translation and rotation invariance in object recognition. Object persistence as a term in psychology sometimes refers to people’s knowledge or belief on the continual existence of an object even when it is occluded and invisible from view. Here, we use it to more generally to denote the temporal persistence of an object in our visual experience. We propose to incorporate the object continuity or persistence constraint in the training of DCNN, and investigate what new abstraction and capability such a network would develop as a consequence. We also evaluate the behaviors of the resulting network to see if they match the data on human perceptual similarity judgment of novel objects in an earlier study (Tenenbaum et al., 2011).
We retrain a DCNN with object persistence constraints, using rendered 3D objects. We call this retrained network Object Persistence Net (OPnet). During training, we utilize a Siamese network architecture for incorporating object persistence constraints into the network. We demonstrated that multi-view association training with a relatively small set of objects directly affects similarity judgment across many classes of objects, including novel objects that the network has not seen before. Our contribution is to demonstrate the surprising transfer of learning of similarity judgment to untrained classes of objects and a variety of completely artificial novel objects. We analyzed the view-manifold fine-tuned with object persistence constraints to understand what changes have taken place in the feature representation of the OPnet that has resulted in the development of this remarkable transfer of perceptual similarity judgment to novel objects.
Creating large sets of human labeled data on object similarity judgement is expensive. There has been a recent trend in exploring inherent information as supervisory signal, including using cycle consistency for learning dense correspondance(Zhou et al., 2015), camera motion for foreground segmentation(Zeng et al., 2016) and context information(Doersch et al., 2015). Among these, most related to our study is the work of Wang & Gupta (2015) utilizing visual tracking results as supervisory signals, which is an object persistence or continuity assumption, to learn deep networks without explicit object labels. While the tracked patches can be partly regarded as multi-view images, the changes in views tend to be very limited. In comparison, we used graphics rendered multi-view images as object persistency constraint. Such a clean setup is necessary for us to study the effect of object persistency constraint on novel objects, as well as the transferability of view-manifold learning to similarity perception.
Recent approaches in representation learning of 3D shapes are also related to our work. Generative models such as (Wu et al., 2016) and (Tatarchenko et al., 2015) learn a vector representation for generation of 3D shapes. Other approaches learn an embedding space for multi-view object retrieval
(Guo et al., 2016) or for cross-view image and shape retrieval(Li et al., 2015). While these works explored training with multi-view images, they did not constrain the view points in a continuous way and most importantly, the transferability to judgement of novel objects of novel classes were not studied. We evaluate the performance of the approach with Li et al. (2015) in our tasks for comparison. That approach learned an embedding space of 3D shapes and used CNN for image embedding for the purpose of image purification.
2 APPROACH AND METHODS
We take a standard CNN (AlexNet), that has already learned good feature representations for object classification, and retrain the network in a Siamese triplet architecture with object persistence constraints using multi-view images rendered from a set of 3D object models in ShapeNet.
2.1 OBJECT PERSISTENT NET (OPNET)
To study the impact of object persistence constraint in the development of perceptual similarity judgment, OPnet utilizes a Siamese triplet architecture. This triplet architecture can be visualized as three baseline CNN towers that share the same parameters (Figure 1). In implementation, it is just one single CNN applied to three images, two of which are considered more “similar” than the third “different” one. Conceptually, our OPnet tries to bring the feature representations of the two “similar” images together, and drive apart the representation corresponding to third “different” image. The architecture and the initial weights of the baseline CNN is same as those of of AlexNet trained on ImageNet (Deng et al., 2009). To train our OPnet with triplet input (Xi, X+i , X − i ), we present two views of the same 3D object to two base networks as (Xi, X+i ), and a view of a different object to the third base network as X−i . Object persistence means that given (Xi, X + i , X − i ), we try to push the representations for views of the same object (Xi, X+i ) to be close and make them away from the representation for the different object X−i . We minimize the loss function with a hinge loss term:
min W
λ 2 ‖W ‖22 + N∑ i=1 max{0, D(Xi, X+i )−D(Xi, X − i ) +M},
D(X1, X2) = 1− f(X1) · f(X2)
‖ f(X1) ‖ · ‖ f(X2) ‖ ,
where λ is the weight decay andW denotes the weights of the network. f(·) is the CNN representation output as a function of an input image, andM denotes the margin parameter. The margin is a threshold
to decide whether the two views are considered similar or not. The higher the margin, the more we are forcing the network to develop a uniform representation for multiple views of the same object, relative to views of another object. D is the cosine distance function for a pair of features.
The different objects in principle could be from the same category or from different categories. During training, we constrain the “different object” to be another 3D object from the same category to push apart more forcefully the feature representations of objects from the same category, resulting in view-invariant object discrimination within the same category. We expect the result of this training to create a view-manifold for each individual object—views within the same manifold are considered to be “similar” and closer together because they belong to the same object.
2.2 DISTANCE METRIC LEARNING
Our Siamese triplet approach transforms the view-manifold of the original baseline network, so that different views of the same object are considered similar and become closer in the feature representation space. Thus, it can be viewed as a form of distance metric learning (DML), which is a set of methods that learn a transformation from the input space to a feature space. The Siamese network has been a popular distance metric learning method, used in signature verification (Bromley et al., 1993), learning invariant mapping (Hadsell et al., 2006), face verification (Chopra et al., 2005), unsupervised learning (Wang & Gupta, 2015) or image similarity ranking (Wang et al., 2014). In these works, the definition of similarity for DML comes from the semantic labeling like class label. In our work, the similarity is defined by the object persistence constraints, obtained during the rendering of 3D models and providing a continuous trajectory for each single object. Besides, the large variation of the 2D appearance induced by 3D rotation prevents our network from learning trivial global templates, but induces it to learn features that are more generalized and thus transferable more easily to novel objects.
DCNNs, such as AlexNet, pre-trained on large dataset, have developed useful feature representations that can be fine-tuned for other specific tasks (Donahue et al., 2014; Qian et al., 2015; Karpathy et al., 2014). However, the pre-training of DCNN involves class labels as teaching signals. During pretraining, the network learns to throw away much information to extract invariants for classification. On the other hand, DML approaches are able to develop feature representations that preserve more fine-grained features, as well as intra- and inter-class variations.
2.3 RENDERING MULTI-VIEW DATASETS FOR SIMILARITY JUDGEMENT TRAINING
To allow the network to learn features under the object persistence constraints and develop a similarity judgment that can transfer, we create one set of data for training and five sets of novel objects for testing of the transferability. To focus our study on the network’s ability to perceive 3D spatial relations and features of individual objects, we grayscale our images during rendering to eliminate the impact of color. For the same reason, we do not add any backgrounds.
We render multi-view images of individual objects from 7K 3D CAD models of objects in ShapeNet (Chang et al., 2015). The 7K models belong to 55 categories, such as cars and chairs. For each model, we render 12 different views by rotating the cameras along the equator from a 30◦ elevation angle and taking photos of the object at 12 equally separated azimuthal angles (see Fig. 1). We use the rendering pipeline in Blender, an open source 3D graphics software, with a spotlight that is static relative to the camera.
For training, we sample 200 object models from 29 categories of ShapeNet. 20 of these object models from each category are saved for cross validation. For testing, we make the assumptions that (1) views of the same object are perceived to be more similar when compared to views of a different object, and (2) views of objects in the same category are perceived to be more similar than views of objects from different categories. These assumptions are consistent with findings in earlier studies on similarity judgment in human (Quiroga et al., 2005; Erdogan et al., 2014; Goldstone, 2013). Since we render images based on CAD models, we can control the variations to create a large dataset that can approximate ground-truth data for similarity judgment for our experiments without resorting to large-scale human judgment evaluation. All the objects in the following five test sets are novel objects in the sense that they are not used in training.
Novel instance: Created by rendering additional 20 novel objects from each of the 29 categories used in training the OPnet. This is used to test the transfer of view-manifold learning to novel objects of the same category. The task is not trivial due to the large intra-category variation existing in the ShapeNet.
Novel category: Created by rendering objects from 26 untrained categories. This is a more challenging test of the transfer of view-manifold learning to novel categories.
Synthesized objects: Created by rendering a set of 3D models we synthesized. These are textureless objects with completely novel shapes. The dataset consists of 5 categories, with 10 instances for each category. Within each category, the objects either have similar local parts, or have the same global configuration, based on human judgment. This is an even more challenging test, as these synthesized objects are not in the ImageNet or ShapeNet.
Pokemon Created by 3D models of Pokemon dataset. Pokemons are cartoon characters with certain evolution relationships with each other, which provides an alternative measurement of similarity. This test evaluates the transfer of learning to novel objects with different styles and more complicated textures. We collected 438 CAD models of Pokemon from an online database. We divide these models into 251 categories according to their evolution relationships, with most of these categories containing only 2 to 4 objects. Pokemons of the same category look more similar on average due to their “genetic linkage”.
Tenenbaum objects This test set contains novel objects from Tenenbaum et al. (2011), where the ground truth is based on human similarity judgment.
2.4 SIMILARITY JUDGMENT EVALUATION
The similarity score between a query image and a candidate image is computed as 1 minus the cosine distance of the feature representations of the query and candidate pair, and higher score means higher similarity. Given a test set containing objects of multiple categories, we evaluate the OPnet via two retrieval tasks: object instance retrieval and categorical retrieval. In the object instance retrieval task, for each image P containing object O of category C in the test set, the network is asked to rank all other images in C, such that images for O should have higher similarity score than images for other objects in C. In the categorical retrieval task, for each image P of category C, the network is asked to rank all other images, such that images in category C should have higher score than images not in C. Here we are indirectly utilizing the human perception information, as categories are defined by human perception based on their similarity in shapes or functions.
2.5 IMPLEMENTATION DETAILS
We use Caffe (Jia et al., 2014) for training the networks. The base network of the OPnet is modified from the AlexNet architecture, where we drop the last fully connected layer (fc8) and replace the softmax loss with our triplet hinge loss. The network is initialized by weights pre-trained on ImageNet. The objective is optimized using mini-batch stochastic gradient descent (SGD) and we fine-tune the network for all layers. For each pair of positive example (X,X+), we select two hard negative examples X− which give the highest loss (similar in (Wang & Gupta, 2015)) and another two randomly from within the mini-batch. Starting with a learning rate of 0.01, we decrease it by a factor of 10 every 8K iterations and with a momentum of 0.9. We stop the training at 20K iterations. Weight decay is set to 0.0005. We set the margin parameter M to 0.1 by cross validation.
3 EXPERIMENTAL RESULTS
We compare HoG feature representation (Dalal & Triggs, 2005) and four deep learning networks: 1) OPnet, 2) AlexNet pre-trained on ImageNet, 3) An AlexNet fine-tuned for classification on ShapeNet data, denoted as “AlexNetFT”, 4) The joint embedding model by Li et al. (2015). In AlexNetFT, we replace the original fc8 layer with a fully connected layer with 29 output units and fine-tune the last two fully connected layers (fc7, fc8) with cross-entropy loss. The AlexNetFT model is trained with the same data we used for training the OPnet. The joint embedding model was pre-trained on 6700 shapes in the chair category of ShapeNet. For the first three deep models, we use the fc7 layer as the feature representation and cosine distance to compute distance between feature representations. We
also show results based on AlexNet feature representation both in terms of Eculidean distance and cosine distance measures, denoted as AlexNet+EcuDis and AlexNet+CosDis. Comparison of feature representations from different layers are shown in Appendix B. We show the results for the instance retrieval task in Figure 2 and Table 1. The precision measure reflects the accuracy of the model’s similarity judgment, with the two assumptions given in section 2.3.
On similarity judgment of novel objects from both the trained and untrained categories, OPnet significantly outperforms AlexNet and AlexNetFT, with an increased Mean Average Precision of at least 23%. The improvement is due to OPnet’s gains in ability in discriminating different objects inside one category regardless of their viewpoints, while recognizing different views of the objects to be similar. For novel shapes in artificial synthesized objects and Pokemons, OPnet still shows an increased MAP of at least 6% (or 15% decreased error rate for the Pokemon test). This shows that the similarity judgment resulting from view manifold learning is valid not only for the trained objects or just to the objects in the same data set, but generalizable to other classes of objects. This suggests the learned feature representations are more abstract and general, allowing the transfer of the learning to substantially different datasets and novel objects, to a degree that is not well known or well studied in computer vision.
We compare OPnet with the joint embedding approach on the chair category of ShapeNet, shown in Figure 2b. Both networks are trained with the chair category and are tested on novel chairs. OPnet outperforms the joint embedding approach by a large margin, showing that a better instance level
discrimination is achieved using object persistence training, compared to using known shapes as anchor points for image embedding. Furthermore, because the joint embedding approach would need to be trained for each specific category, it does not perform well on novel categories.
When we fine-tuned AlexNet for classification of the 29 trained categories, the resulting AlexNetFT’s feature representation actually performs the worst, compared to OPnet and the original AlexNet, on the instance similarity judgment or retrieval tasks. When a network is trained to perform classification, it learns to ignore subtle differences among objects in the same category. The fewer categories a network is trained on, the more the instance level similarity judgment will be compromised. This loss of the generality of its feature representation compromises its transferability to novel objects in other classes.
We notice that the performance gain for the OPnet is most significant in the ShapeNet dataset and the gap becomes small for the synthesized and Pokemon dataset. This shows OPnet’s certain overfitting to the bias in ShapeNet, as the synthesized object dataset contains textureless objects and Pokemon dataset contains mainly human-like characters that are not in ShapeNet.
Categorical retrieval provides another measure of the network’s performance in similarity judgment. In this test, we randomly sample 20 categories each from the novel instance test and the novel category test, with 20 object instances drawn from each category. For the synthesized object test set, we test all 5 categories and each with 10 instances. For each instance, a single random view is provided. The results are shown in Figure 3. Despite the fact that AlexNet knows more about the semantic features of each category, our OPnet still achieves comparable results. OPnet here shows an improved ability in similarity judgment at the categorical level. On our artificially synthesized object dataset, where all three networks have no prior experience, OPnet performs better than AlexNet. AlexNetFT performs extremely well on trained categories likely because it is overfitted to the limited trained objects, even though it uses the same amount of data. This overfitting problem shows that training with only class labels might not preserve the essential information to develop transferable general feature and abstract feature representation, especially with limited training dataset.
3.1 CORRELATION WITH HUMAN PERCEPTION
Using the novel objects from Tenenbaum et al. (2011), we are able to compare our networks with human similarity perception. We collect 41 images from the paper, one image per object. A pairwise similarity matrix is calculated based on the cosine distance of their feature representations. We can then perform hierarchical agglomerative clustering to obtain a tree structure, using the Nearest Point Algorithm. That is, for all points i in cluster u and points j in cluster v, the distance of the two clusters are calculated by dist(u, v) = min(D(u[i], v[j])), where D(·) is the cosine distance function. We merge two clusters with the shortest distance successively to construct the tree. The tree based on human perception is constructed by giving human subjects all the images and asking them to merge two clusters that are most similar each time, similar to the hierarchical agglomerative clustering algorithm. Results are shown in Figure 4.
In order to quantitatively measure the similarity between the trees output by neural networks and the one based on human perception, we calculate the Cophenetic distances on the tree for each pair of
object. For object i and j, the Cophenetic distances ti,j are defined as ti,j = dist(u, v), i ∈ u, j ∈ v, where u,v are clusters connected by U-link. Finally, we can evaluate the similarity of the two trees by calculating the Spearman’s rank correlation coefficient. In the experiment, the Spearman correlation is 0.460 between AlexNet and the human perception and 0.659 between OPnet and the human perception, meaning that our OPnet, trained with object persistence constraints on a relatively small set of objects, automatically yielded a higher match to the human perceptual similarity data. This finding provides some support to our conjecture that object persistence might play an important role in shaping human similarity judgment.
3.2 STRUCTURES OF THE VIEW MANIFOLD
We study the feature representations in these networks and their transformation induced by the object persistence constraints to understand how the changes in similarity judgment performance come about. As our network uses cosine distance in the feature space as similarity measure, we study how this measure changes in the view-manifold of the same object and between views of different objects.
We first visualize the pairwise similarity distance matrix of AlexNet and OPnet in Figure 5. We randomly choose 5 objects from the cabinet category for illustration. Each object has 12 views that the network has never seen before. Images are arranged first by different object instances (in columns) then by views (in rows). Many properties of the view manifolds are revealed. First, for the matrix of OPnet, we can see clearly five dark blocks formed in the diagonal, each standing for the strong similarity (small distance) among the different views of the same cabinet. The dark block means that OPnet is associating different views of the same object together, reducing intra-object distance relative to inter-object distance. In this way, the similarity judgment of the OPnet becomes more viewpoint independent. On the other hand, the similarity matrix of AlexNet shows a variety of patterns across all objects within the same category. A closer look at these patterns suggests that AlexNet first forms groups by certain views (e.g. side-views), and then by objects, resulting in a more viewpoint dependent similarity measure that is poor in discriminating objects within a category. Second, even though OPnet groups different views together, the view-manifold has not degenerated into a single point. Certain patterns can be seen inside each dark block of OPnet’s matrix, forming a hierarchical structure: different views of the same object are more similar to each other than to another object and some rotations in angle are considered more similar than others. To illustrate how the view manifolds have contracted but not completely degenerated, we randomly sample objects from the novel instance test set and use TSNE (Maaten & Hinton, 2008) to plot them in 2D, as shown in Figure 6. We can see clearly that different views of the same object are considered more similar in the feature space, and objects form tight and distinct clusters. We borrow a measurement from Linear
Discriminant Analysis (LDA) to evaluate how tightly different views of the same object are clustered together, relative to the distance among different objects within the same category. Formally, let S be the set of all the objects inside one category and c be the set of all views for one object, x̄ be the center of all image features, and µc be the center for the object c. We then calculate the score for category i using the following equation:
Scorei = σinter instance σintra instance =
1 |Si| ∑ c ||µc − x̄||2
1 |Si| ∑ c∈Si 1 |c| ∑ x∈c ||x− µc||2
We then average over all the categories to get a score for each network. The higher the score is, the larger the inter-object distance is compared to intra object distance and the more closely different views of the same object are grouped together. In the experiment with the novel instance test set, OPnet’s score is 0.535 whereas AlexNet’s is 0.328, showing the different views of the same object are more similar than that between different objects due to the object persistence constraint.
4 CONCLUSION AND DISCUSSION
In this work, we fine-tune AlexNet with object persistence constraints in the framework of distance metric learning with a Siamese triplet. This fine-tuning modifies the view-manifold of the object
representation, bringing closer together the representations of an object in different views, driving apart representations of different objects in the same category, resulting in better intra-categorical object recognition, without compromising inter-categorical discrimination. We investigated whether this view-manifold learning results in an improvement in the network’s ability to recognize the similarity of novel objects that have never been seen before by performing instance and categorical image retrieval on artificial novel objects or novel object classes, including a set tested in human similarity judgment. Interestingly, we find that AlexNet, with its rich feature representations, already perform similarity judgement significantly above chance, in the sense that different views of the same object are considered more similar to the views of another object in the same category, or objects in the same category are considered to be more similar than objects in different categories. Fine-tuning with the object persistence constraint significantly improves this ”similarity judgement” among a variety of novel objects, suggesting the view manifold learning in the OPnet is accompanied by feature embeddings with more general and abstract attributes that are transferable, likely at the level of local object parts.
From a technical point of view, our OPnet performs better than earlier approaches (Li et al., 2015) in instance and categorical retrieval of novel objects. We have tested our approach with real image database (Geusebroek et al., 2005) and found it only yields a slight improvement over AlexNet. That database contains 1000 objects with different views but without categorical labels. OPnet’s superiority over AlexNet lies in its better discrimination of objects within the same category. When objects are not organized in categories, i.e. when each object is essentially treated as a category, OPnet loses its advantages. In addition, there are more complex variations such as lighting and scale in real scene environments that our current OPnet has not considered. We plan to develop this model to discount additional nuisance variables and to develop or find database to explore the transferability of its view-manifold learning in more general settings.
Our work was motivated by our hypothesis that object persistence/continuity constraint in our visual experience might play a role in the development of neural representations that shape our similarity judgement of objects that we have not seen before. The fact that fine-tuning AlexNet with this additional constraint automatically yields a new view-manifold that match human similarity judgment data better than AlexNet lends some support to our hypothesis. However, more extensive testing with human perception ground-truth will be needed to fully confirm our hypothesis.
ACKNOWLEDGMENTS
Xingyu Lin and Hao Wang were supported by the PKU-CMU summer internship program. This work is supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DoI/IBC) contract number D16PC00007. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/IBC, or the U.S. Government.
We thank Kalina Ko for helping us to construct part of the synthesized object database.
APPENDIX A EXAMPLES OF SOME TOP RANKING RESULTS
APPENDIX B INSTANCE RETRIEVAL RESULTS USING FEATURES FROM DIFFERENT LAYERS
As shown in many literatures (Massa et al., 2015; Aubry & Russell, 2015), features from different layers sometimes perform differently for a given task. For the instance retrieval task on the novel instance dataset of the ShapeNet, we compare OPnet and AlexNet using features from different layers, as shown in Figure 8. The accuracy of AlexNet is pretty flat up to conv3, and then keeps increasing until layer fc8 where the feature becomes categorical probability and not appropriate for instance level discrimination. On the other hand, the object persistence training gives a significant increase in accuracy in fully connected layers.

I think learning a deep feature representation that is supervised to group dissimilar views of the same object is interesting. The paper isn't technically especially novel but that doesn't bother me at all. It does a good job exploring a new form of supervision with a new dataset. I'm also not bothered that the dataset is synthetic, but it would be good to have more experiments with real data, as well.
I think the paper goes too far in linking itself to human vision. I would prefer the intro not have as much cognitive science or neuroscience. The second to fourth paragraphs of the intro in particular feels like it over-stating the contribution of this paper as somehow revealing some truth about human vision. Really, the narrative is much simpler -- "we often want deep feature representations that are viewpoint invariant. We supervise a deep network accordingly. Humans also have some capability to be viewpoint invariant which has been widely studied [citations]". I am skeptical of any claimed connections bigger than that.
I think 3.1 should not be based on tree-to-tree distance comparisons but instead based on the entire matrix of instance-to-instance similarity assessments. Why do the lossy conversion to trees first? I don't think "Remarkably" is justified in the statement "Remarkably, we found that OPnets similarity judgement matches a set of data on human similarity judgement, significantly better than AlexNet"
I'm not an expert on human vision, but from browsing online and from what I've learned before it seems that "object persistence" frequently relates to the concept of occlusion. Occlusion is never mentioned in this paper. I feel like the use of human vision terms might be misleading or overclaiming.