IMPLICIT REASONET: MODELING LARGE-SCALE STRUCTURED RELATIONSHIPS WITH SHARED MEM- ORY
Recent studies on knowledge base completion, the task of recovering missing relationships based on recorded relations, demonstrate the importance of learning embeddings from multi-step relations. However, due to the size of knowledge bases, learning multi-step relations directly on top of observed instances could be costly. In this paper, we propose Implicit ReasoNets (IRNs), which is designed to perform large-scale inference implicitly through a search controller and shared memory. Unlike previous work, IRNs use training data to learn to perform multi-step inference through the shared memory, which is also jointly updated during training. While the inference procedure is not operating on top of observed instances for IRNs, our proposed model outperforms all previous approaches on the popular FB15k benchmark by more than 5.7%.
1 INTRODUCTION
Knowledge bases such as WordNet (Fellbaum, 1998), Freebase (Bollacker et al., 2008), or Yago (Suchanek et al., 2007) contain many real-world facts expressed as triples, e.g., (Bill Gates, FounderOf, Microsoft). These knowledge bases are useful for many downstream applications such as question answering (Berant et al., 2013; Yih et al., 2015) and information extraction (Mintz et al., 2009). However, despite the formidable size of knowledge bases, many important facts are still missing. For example, West et al. (2014) showed that 21% of the 100K most frequent PERSON entities have no recorded nationality in a recent version of Freebase. We seek to infer unknown relations based on the observed triples. Thus, the knowledge base completion (KBC) task has emerged an important open research problem (Nickel et al., 2011).
Neural-network based methods have been very popular for solving the KBC task. Following Bordes et al. (2013), one of the most popular approaches for KBC is to learn vector-space representations of entities and relations during training, and then apply linear or bi-linear operations to infer the missing relations at test time. However, several recent papers demonstrate limitations of prior approaches relying upon vector-space models alone. By themselves, there is no straightforward way to capture the structured relationships between multiple triples adequately (Guu et al., 2015; Toutanova et al., 2016; Lin et al., 2015a). For example, assume that we want to fill in the missing relation for the triple (Obama, NATIONALITY, ?), a multi-step search procedure might be needed to discover the evidence in the observed triples such as (Obama, BORNIN, Hawaii) and (Hawaii, PARTOF, U.S.A). To address this issue, Guu et al. (2015); Toutanova et al. (2016); Lin et al. (2015a) propose different approaches of injecting structured information by directly operating on the observed triplets. Unfortunately, due to the size of knowledge bases, these newly proposed approaches suffer from some limitations, as most paths are not informative for inferring missing relations, and it is prohibitive to consider all possible paths during the training time with expressive models.
In this paper, we take a different approach from prior work on KBC by addressing the challenges of performing large-scale inference through the design of search controller and shared memory. Our inference procedure centers around the search controller, which only operates on the shared memory instead of directly manipulating the observed triples in knowledge base. IRNs use training data to
∗Equal contribution.
learn to perform multi-step inference through the shared memory. First, input module generates a representation of the query. Then, the search controller repeatedly interacts with the shared memory and checks the termination gate. After each iteration, if the termination condition is met, the model stops the search process and calls the output module to generate a prediction. The shared memory is designed to store key information about the overall structures it learned during training, and hence the search controller only needs to access the shared memory instead of operating on the observed triples.
There are several advantages of using IRNs. First, the cost of inference can be controlled because the search controller only needs to access the shared memory. Second, all the modules, including the search controller and memory, are jointly trained, and hence alleviate the needs to inject structured relationships between instances manually. Finally, we can easily extend IRNs to other tasks that require modeling structured relationships between instances by switching the input and output modules.
The main contributions of our paper are as follows:
• We propose Implicit ReasoNets (IRNs), which use a shared memory guided by a search controller to model large-scale structured relationships implicitly.
• We evaluate IRNs and demonstrate that our proposed model achieves the state-of-the-art results on the popular FB15k benchmark, surpassing prior approaches by more than 5.7%.
• We analyze the behavior of IRNs for shortest path synthesis. We show that IRNs outperform a standard sequence-to-sequence model and execute meaningful multi-step inference.
2 REASONET FOR IMPLICIT INFERENCE
In this section, we describe the general architecture of IRNs in a way that is agnostic to KBC. IRNs are composed of four main components: an input component, an output component, a shared memory, and a search controller, as shown in Figure 1. In this section, we briefly describe each component.
Input/Output Modules: These two modules are task-dependent. The input module takes a query and converts the query into a vector representation q. The output module is a function fo, which converts the hidden state received from the search controller (s) into an output O. We optimize the whole
model using the output prediction O with respect to a ground-truth target using a task-specified loss function.
Shared Memory: The shared memory is denoted as M . It consists of a list of memory vectors, M = {mi}i=1...I , where mi is a fixed dimensional vector. The memory vectors are randomly initialized and automatically updated through back-propagation. The shared memory component is shared across all instances.
Search Controller: The search controller is a recurrent neural network and controls the search process by keeping internal state sequences to track the current search process and history. The search controller uses an attention mechanism to fetch information from relevant memory vectors in M , and decides if the model should output the prediction or continue to generate the next possible output.
• Internal State: The internal state of the search controller is denoted as S, which is a vector representation of the search process. The initial state s1 is usually the vector representation of the input vector q. The internal state at t-th time step is represented by st. The sequence of internal states is modeled by an RNN: st+1 = RNN(st, xt; θs).
• Attention to memory: The attention vector xt at t-th time step is generated based on the current internal state st and the shared memory M : xt = fatt(st,M ; θx). Specifically, the attention score at,i on a memory vector mi given a state st is computed as at,i = softmaxi=1,...,|M |λ cos(W1mi,W2st), where λ is set to 10 in our experiments and the weight matrices W1 and W2 are learned during training. The attention vector xt can be written as xt = fatt(st,M ; θx) = ∑|M | i at,imi.
• Termination Control: The terminate gate produces a stochastic random variable according to the current internal state, tt ∼ p(·|ftc(st; θtc))). tt is a binary random variable. If tt is true, the IRN will finish the search process, and the output module will execute at time step t; otherwise the IRN will generate the next attention vector xt+1 and feed into the state network to update the next internal state st+1. In our experiments, the termination variable is modeled by a logistical regression: ftc(st; θtc) = sigmoid(Wtcst + btc), where the weight matrix Wtc and bias vector btc are learned during training.
Compared IRNs to Memory Networks (MemNN) (Weston et al., 2014; Sukhbaatar et al., 2015; ?) and Neural Turing Machines (NTM) (Graves et al., 2014; 2016), the biggest difference between our model and the existing frameworks is the search controller and the use of the shared memory. We build upon our previous work (Shen et al., 2016) for using a search controller module to dynamically perform a multi-step inference depending on the complexity of the instance. MemNN and NTM explicitly store inputs (such as graph definition, supporting facts) in the memory. In contrast, in IRNs, we do not explicitly store all the observed inputs in the shared memory. Instead, we directly operate on the shared memory, which modeling the structured relationships implicitly. We randomly initialize the memory and update the memory with respect to task-specific objectives. The idea of exploiting shared memory is proposed by Munkhdalai & Yu (2016) independently. Despite of using the same term, the goal and the operations used by IRNs are different from the one used in Munkhdalai & Yu (2016), as IRNs allow the model to perform multi-step for each instance dynamically.
2.1 STOCHASTIC INFERENCE PROCESS
The inference process of an IRN is as follows. First, the model converts a task-dependent input to a vector representation through the input module. Then, the model uses the input representation to initialize the search controller. In every time step, the search controller determines whether the process is finished by sampling from the distribution according to the terminate gate. If the outcome is termination, the output module will generate a task-dependent prediction given the search controller states. If the outcome is continuation, the search controller will move on to the next time step, and create an attention vector based on the current search controller state and the shared memory. Intuitively, we design whole process by mimicking a search procedure that iteratively finds its target through a structure and output its prediction when a satisfying answer is found. The detailed inference process is described in Algorithm 1.
The inference process of an IRN is considered as a Partially Observable Markov Decision Process (POMDP) (Kaelbling et al., 1998) in the reinforcement learning (RL) literature. The IRN produces
Algorithm 1: Stochastic Inference Process in an IRN Input :Randomly initialized shared memory M ; Input vector q; Maximum step Tmax Output :Output vector o
1 Define s1 = q; t = 1; 2 Sample tt from the distribution p(·|ftc(st; θtc)); 3 if tt is false, go to Step 4; otherwise Step 7; 4 Generate an attention vector xt = fatt(st,M ; θx); 5 Update the internal state st+1 = RNN(st, xt; θs); 6 Set t = t+ 1; if t < Tmax go to Step 2; otherwise Step 7; 7 Generate output ot = fo(st; θo); 8 Return o = ot;
the output vector oT at the T -th step, which implies termination gate variables t1:T = (t1 = 0, t2 = 0, ..., tT−1 = 0, tT = 1), and then takes prediction action pT according to the probability distribution given oT . Therefore, the IRN learns a stochastic policy π((t1:T , pT )|q; θ) with parameters θ to get a distribution over termination actions, and over prediction actions. The termination step T varies from instance to instance. The parameters of the IRN θ are given by the parameters of the embedding matrices W for the input/output module, the shared memory M , the attention network θx, the search controller RNN network θs, the output generation network θo, and the termination gate network θtc. The parameters θ = {W,M, θx, θs, θo, θtc} are trained to maximize the total expected reward that the IRN when interacting with the environment. The expected reward for an instance is defined as:
J(θ) = Eπ(t1:T ,pT ;θ) [ T∑ t=1 rt ]
The reward can only be received at the final termination step when a prediction action pT is performed. The rewards on intermediate steps are zeros, {rt = 0}t=1...T−1. We employ the approach from our previous work (Shen et al., 2016), REINFORCE (Williams, 1992) based Contrastive Reward method, to maximize the expected reward. The gradient of J can be written as:
∇θJ(θ) = ∑
(t1:T ,pT )∈A† π(t1:T , pT ; θ)
[ ∇θlogπ(t1:T , pT ; θ)(
rT bi − 1)
]
where A† is all the possible episodes, the baseline bi = ∑
(t1:T ,pT )∈A† π(t1:T , pT ; θ)rT is the expected reward on the |A†| episodes for the i-th training instance.
3 APPLYING IRNS TO KNOWLEDGE BASE COMPLETION
The goal of KBC tasks (Bordes et al., 2013) is to predict a head or a tail entity given the relation type and the other entity, i.e. predicting h given (?, r, t) or predicting t given (h, r, ?), where ? denotes the missing entity. For a KBC task, the input to our model is a subject entity (a head or tail entity) and a relation. The task-dependent input module first extracts the embedding vectors for the entity and relation from an embedding matrix. We then represent the query vector q for an IRN as the concatenation of the two vectors. We randomly initialize the shared memory component. At each step, a training triplet is processed through the model by Algorithm 1, where no explicit path information is given. The IRN updates the shared memory implicitly with respect to the objective function. For the task dependent output module, we use a nonlinear projection to project the search controller state into an output vector o: fo(st; θo) = tanh(Wost + bo), where the Wo and bo are the weight matrix and bias vector, respectively. We define the ground truth target (object) entity embedding as y, and use the L1 distance measure between the output o and target entity y, namely d(o, y) = |o− y|1. We sample a set of incorrect entity embeddings N = {y−i } |N | i=1 as negative examples. The probability of
selecting a prediction ŷ ∈ D can be approximated as
p(ŷ|o) = exp(−γd(o, ŷ))∑ yk∈D exp(−γd(o, yk))
where D = N ∪ {y}. We set |N | and γ to 20 and 5, respectively, for the experiments on FB15k and WN18 datasets. The IRN performs a prediction action pT on selecting ŷ with probability p(ŷ|o) . We define the reward of the prediction action as one if the ground truth entity is selected, and zero otherwise.
4 EXPERIMENTAL RESULTS
In this section, we evaluate the performance of our model on the benchmark FB15k and WN18 datasets for KBC tasks (Bordes et al., 2013). These datasets contain multi-relations between head and tail entities. Given a head entity and a relation, the model produces a ranked list of the entities according to the score of the entity being the tail entity of this triple. To evaluate the ranking, we report mean rank (MR), the mean of rank of the correct entity across the test examples, and hits@10, the proportion of correct entities ranked in the top-10 predictions. Lower MR or higher hits@10 indicates a better prediction performance. We follow the evaluation protocol in Bordes et al. (2013) to report filtered results, where negative examples N are removed from the dataset. In this case, we can avoid some negative examples being valid and ranked above the target triplet.
We use the same hyper-parameters of our model for both FB15k and WN18 datasets. Entity embeddings (which are not shared between input and output modules) and relation embedding are both 100-dimensions. We use the input module and output module to encode subject and object entities, respectively. There are 64 memory vectors with 200 dimensions each, initialized by random vectors with unit L2-norm. We use single-layer GRU with 200 cells as the search controller. We set the maximum inference step of the IRN to 5. We randomly initialize all model parameters, and use SGD as the training algorithm with mini-batch size of 64. We set the learning rate to a constant number, 0.01. To prevent the model from learning a trivial solution by increasing entity embeddings norms, we follow Bordes et al. (2013) to enforce the L2-norm of the entity embeddings as 1. We use hits@10 as the validation metric for the IRN. Following the work (Lin et al., 2015a), we add reverse relations into the training triplet set to increase the training data.
Following Nguyen et al. (2016), we divide the results of previous work into two groups. The first group contains the models that directly optimize a scoring function for the triples in a knowledge base without using extra information. The second group of models make uses of additional information from multi-step relations. For example, RTransE (García-Durán et al., 2015) and PTransE (Lin et al., 2015a) models are extensions of the TransE (Bordes et al., 2013) model by explicitly exploring multi-step relations in the knowledge base to regularize the trained embeddings. The NLFeat model (Toutanova et al., 2015) is a log-linear model that makes use of simple node and link features.
Table 1 presents the experimental results. According to the table, our model significantly outperforms previous baselines, regardless of whether previous approaches use additional information or not. Specifically, on FB15k, the MR of our model surpasses all previous results by 12, and our hit@10 outperforms others by 5.7%. On WN18, the IRN obtains the highest hit@10 while maintaining similar MR results compared to previous work.1
To better understand the behavior of IRNs, we report the results of IRNs with different memory sizes and different Tmax on FB15K in Table 2. We find the performance of IRNs increases significantly if the number of inference step increases. Note that an IRN with Tmax = 1 is the case that an IRN without the shared memory. Interestingly, given Tmax = 5, IRNs are not sensitive to memory sizes. In particular, larger memory always improves the MR score, but the best hit@10 is obtained by |M | = 64 memory vectors. A possible reason is that the best memory size is determined by the complexity of the tasks.
We analyze hits@10 results on FB15k with respect to the relation categories. Following the evaluation in Bordes et al. (2013), we evaluate the performance in four types of relation: 1-1 if a head entity
1Nguyen et al. (2016) reported two results on WN18, where the first one is obtained by choosing to optimize hits@10 on the validation set, and second one is obtained by choosing to optimize MR on the validation set. We list both of them in Table 1.
can appear with at most one tail entity, 1-Many if a head entity can appear with many tail entities, Many-1 if multiple heads can appear with the same tail entity, and Many-Many if multiple head entities can appear with multiple tail entities. The detailed results are shown in Table 3. The IRN significantly improves the hits@10 results in the Many-1 category on predicting the head entity (18.8%), the 1-Many category on predicting the tail entity (16.5%), and the Many-Many category (over 8% in average).
To analyze the behavior of IRNs, we pick some examples for the tail entity prediction in Table 4. Interestingly, we observed that the model can gradually increase the ranking score of the correct tail entity during the inference process.
5 ANALYSIS: APPLYING IRNS TO A SHORTEST PATH SYNTHESIS TASK
We construct a synthetic task, shortest path synthesis, to evaluate the inference capability over a shared memory. The motivations of applying our model to this task are as follows. First, we want to evaluate IRNs on another task requiring multi-step inference. Second, we select the sequence generation task so that we are able to analyze the inference capability of IRNs in details.
In the shortest path synthesis task, as illustrated in Figure 2, a training instance consists of a start node and an end node (e.g., 215 493) of an underlying weighted directed graph that is unknown to models. The output of each instance is the shortest path between the given start and end nodes of the underlying graph (e.g., 215→ 101→ 493). Specifically, models can only observe the start-end node
pairs as input and their shortest path as output. The whole graph is unknown to the models and the edge weights are not revealed in the training data. At test time, a path sequence is considered correct if it connects the start node and the end node of the underlying graph, and the cost of the predicted path is the same as the optimal path.
Note that the task is very difficult and cannot be solved by dynamic programming algorithms since the weights on the edges are not revealed to the algorithms or the models. To recover some of the shortest paths at the test time, the model needs to infer the correct path from the observed instances. For example, assume that we observe two instances in the training data, “A D: A→ B → G→ D” and “B E: B → C → E”. In order to answer the shortest path between A and E, the model needs to infer that “A→ B → C → E” is a possible path between A and E. If there are multiple possible paths, the model has to decide which one is the shortest one using statistical information.
In the experiments, we construct a graph with 500 nodes and we randomly assign two nodes to form an edge. We split 20,000 instances for training, 10,000 instances for validation, and 10,000 instances for testing. We create the training and testing instances carefully so that the model needs to perform inference to recover the correct path. We describe the details of the graph and data construction parts in the appendix section. A sub-graph of the data is shown in Figure 2.
For the settings of the IRN, we switch the output module to a GRU decoder for a sequence generation task. We assign reward rT = 1 if all the prediction symbols are correct and 0 otherwise. We use a 64-dimensional embedding vector for input symbols, a GRU controller with 128 cells, and a GRU decoder with 128 cells. We set the maximum inference step Tmax to 5.
We compare the IRN with two baseline approaches: dynamic programming without edge-weight information and a standard sequence-to-sequence model (Sutskever et al., 2014) using a similar parameter size to our model. Without knowing the edge weights, dynamic programming only recovers 589 correct paths at test time. The sequence-to-sequence model recovers 904 correct paths. The IRN outperforms both baselines, recovering 1,319 paths. Furthermore, 76.9% of the predicted paths from IRN are valid paths, where a path is valid if the path connects the start and end node nodes of the underlying graph. In contrast, only 69.1% of the predicted paths from the sequence-to-sequence model are valid.
To further understand the inference process of the IRN, Figure 2 shows the inference process of a test instance. Interestingly, to make the correct prediction on this instance, the model has to perform a fairly complicated inference.2 We observe that the model cannot find a connected path in the first three steps. Finally, the model finds a valid path at the forth step and predict the correct shortest path sequence at the fifth step.
6 RELATED WORK
Link Prediction and Knowledge Base Completion Given that r is a relation, h is the head entity, and t is the tail entity, most of the embedding models for link prediction focus on finding the scoring function fr(h, t) that represents the implausibility of a triple. (Bordes et al., 2011; 2014; 2013; Wang et al., 2014; Ji et al., 2015; Nguyen et al., 2016). In many studies, the scoring function fr(h, t) is linear or bi-linear. For example, in TransE (Bordes et al., 2013), the function is implemented as fr(h, t) = ‖h+ r− t‖, where h, r and t are the corresponding vector representations. Recently, different studies (Guu et al., 2015; Lin et al., 2015a; Toutanova et al., 2016) demonstrate the importance for models to also learn from multi-step relations. Learning from multi-step relations injects the structured relationships between triples into the model. However, this also poses a technical challenge of considering exponential numbers of multi-step relationships. Prior approaches address this issue by designing path-mining algorithms (Lin et al., 2015a) or considering all possible paths using a dynamic programming algorithm with the restriction of using linear or bi-linear models only (Toutanova et al., 2016). Toutanova & Chen (2015) shows the effectiveness of using simple node and link features that encode structured information on FB15k and WN18. In our work, the IRN outperforms prior results and shows that similar information can be captured by the model without explicitly designing features.
2 In the example, to find the right path, the model needs to search over observed instances “215 448: 215→ 101→ 448” and “76 493: 76→ 308→ 101→ 493”, and to figure out the distance of “140→ 493” is longer than “101→ 493” (there are four shortest paths between 101→ 493 and three shortest paths between 140→ 493 in the training set).
Studies such as (Riedel et al., 2013) show that incorporating textual information can further improve the knowledge base completion tasks. It would be interesting to incorporate the information outside the knowledge bases in our model in the future.
Neural Frameworks Sequence-to-sequence models (Sutskever et al., 2014; Cho et al., 2014) have shown to be successful in many applications such as machine translation and conversation modeling (Sordoni et al., 2015). While sequence-to-sequence models are powerful, recent work has shown that the necessity of incorporating an external memory to perform inference in simple algorithmic tasks (Graves et al., 2014; 2016).
7 CONCLUSION
In this paper, we propose Implicit ReasoNets (IRNs), which perform inference over a shared memory that models large-scale structured relationships implicitly. The inference process is guided by a search controller to access the memory that is shared across instances. We demonstrate and analyze the multi-step inference capability of IRNs in the knowledge base completion tasks and a shortest path synthesis task. Our model, without using any explicit knowledge base information in the inference procedure, outperforms all prior approaches on the popular FB15k benchmark by more than 5.7%.
For future work, we aim to further extend IRNs in two ways. First, inspired from Ribeiro et al. (2016), we would like to develop techniques to exploit ways to generate human understandable reasoning interpretation from the shared memory. Second, we plan to apply IRNs to infer the relationships in unstructured data such as natural language. For example, given a natural language query such as “are rabbits animals?”, the model can infer a natural language answer implicitly in the shared memory without performing inference directly on top of huge amount of observed sentences such as “all mammals are animals” and “rabbits are animals”. We believe the ability to perform inference implicitly is crucial for modeling large-scale structured relationships.
ACKNOWLEDGMENTS
We thank Scott Wen-Tau Yih, Kristina Toutanova, Jian Tang and Zachary Lipton for their thoughtful feedback and discussions.
A DETAILS OF THE GRAPH CONSTRUCTION FOR THE SHORTEST PATH SYNTHESIS TASK
We construct the underlying graph as follows: on a three-dimensional unit-sphere, we randomly generate a set of nodes. For each node, we connect its K-nearest neighbors and use the euclidean distance between two nodes to construct a graph. We randomly sample two nodes and compute its shortest path if it is connected between these two nodes. Given the fact that all the sub-paths within a shortest path are shortest paths, we incrementally create the dataset and remove the instances which are a sub-path of previously selected paths or are super-set of previous selected paths. In this case, all the shortest paths can not be answered through directly copying from another instance. In addition, all the weights in the graph are hidden and not shown in the training data, which increases the difficulty of the tasks. We set k = 50 as a default value.

[Summary]
This paper proposes a new way for knowledge base completion which highlights: 1) adopting an implicit shared memory, which makes no assumption about its structure and is completely learned during training; 2) modeling a multi-step search process that can decide when to terminate.
The experimental results on WN18 and FB15k seem pretty good. The authors also perform an analysis on a shortest path synthetic task, and demonstrate that this model is better than standard seq2seq.
The paper is well-written and it is easy to follow.
[Major comments]
I actually do like the idea and am also impressed that this model can work well.
The main concern is that this paper presents too little analysis about how it works and whether it is sensitive to the hyper-parameters, besides that only reporting a final model on WN18 and FB15k.
One key hyper-parameter I believe is the size of shared memory (using 64 for the experiments). I don’t think that this number should be fixed for all tasks, at least it should depend on the KB scale. Could you verify this in your experiments? Would it be even possible to make a memory structure with dynamic size?
The RL setting (stochastic search process) is also one highlight of the paper, but could you demonstrate that how much it does really help? I think it is necessary to compare to the following: remove the termination gate and fix the number of inference steps and see how well the model does? Also show how the performance varies on # of steps?
I appreciate your attempts on the shortest path synthetic task. However, I think it would be much better if you can demonstrate that under a real KB setting. You can still perform the shortest path analysis, but using KB  (e.g., Freebase) entities and relations.
[Minor comments]
I am afraid that the output gate illustrated in Figure 1 is a bit confusing. There should be only one output, depending on when the search process is terminated.


We address the problem of modeling multiple simultaneous time series where the observations are correlated not only inside each series, but among the different series. This problem happens in many domains such as ecology, meteorology, etc. We propose a new dynamical state space model, based on representation learning, for modeling the evolution of such series. The joint relational and temporal dynamics of the series are modeled as Gaussian distributions in a latent space. A decoder maps the latent representations to the observations. The two components (dynamic model and decoder) are jointly trained. Using stochastic representations allows us to model the uncertainty inherent to observations and to predict unobserved values together with a confidence in the prediction.
1 INTRODUCTION
Relational time series, i.e. multiple time series where the observations are correlated both inside each series and between series occur in many domains such as ecology, medicine, biology, earth observation by satellite imagery or local measurements, multimedia or even social data analysis. The correlations between the different observed series can come from a proximity (e.g. earth observation or epidemic diffusion) or from a similarity of behavior (e.g. user traces in social data). In the statistical literature, the modeling of relational time series has been the topic of a dedicated field: spatio-temporal statistics (Cressie & Wikle (2011); Wikle & Hooten (2010)). Different methodologies have been developed for handling a large variety of spatio-temporal phenomena, with an emphasis on the analysis of natural observations like weather prediction, ecology or remote sensing. In the machine learning domain, there exists a vast literature dedicated to sequence or time series prediction. Recently, deep recurrent neural networks have witnessed notable successes in different sequence and time series modeling tasks leading to an increasing number of publications, e.g. (Barbounis et al. (2006); Hsieh et al. (2011); Cao et al. (2012); Hermans & Schrauwen (2013)). Despite a large number of recent developments, the modeling and analysis of relational time series has only attracted a few attention in the field of representation learning. In addition, most of the models are deterministic in the sense that they are trained to learn a fixed mapping for modeling the dynamics of the series.
We propose a new state space model for relational time series able to model the uncertainty at the observation and at the modeling levels. The principle of this approach is to associate each point of a time series to a Gaussian distribution in a latent space, the distribution over the observed values being directly computed from these latent distributions. The model has two main components. One is responsible for the dynamics in the latent space. This component is thus modeling the evolution of the Gaussian distribution considering both the temporal intra-series and the relational inter-series
˚Both authors contributed equally to this work
dependencies. A second component acts as a decoder and maps the latent representations associated with each series to the corresponding observations in the output space.
The contributions of the paper are thus: (i) a new dynamical model for relational time series inspired by representation learning; (ii) a stochastic component for modeling the uncertainties at the observation and dynamic levels
The paper is organized as follows. In Section 2 we introduce some related work on forecasting in time series, representation learning for time series, and recent deep learning works focusing on modeling uncertainty. The model is presented in Section 3 together with four different variants. Section 4 presents experimental results on four datasets, and section 5 concludes this work and gives some perspectives.
2 RELATED WORK
The classical topic of time series modeling and forecasting has given rise to an extensive literature. In statistics, classical linear models include many variations around auto-regressive and moving average models (De Gooijer & Hyndman (2006)). In machine learning, non linear extensions of these models based on neural networks have been proposed as early as the 90s, opening the way to many other non linear models including kernel methods (Muller et al. (99)).
Relational time series have mainly been studied in the field of spatio-temporal statistics (Cressie & Wikle (2011); Wikle & Hooten (2010)). The traditional method first relied on a descriptive approach using the first and second-order moments of the process for modeling the spatio-temporal dependencies. More recently, dynamical state models, where the current state is conditioned on the past have been explored (Wikle (2015)). These models have been considered both for continuous/discrete space and time components. However, the most common way is to consider discrete time, leading to the modeling of time series of spatial processes as we do here. When space is discrete, the model comes down to a general vectorial autoregressive formulation. These models face a curse of dimensionality in the case of a large number of sources. Different strategies have been adopted to solve this problem such as embedding the spatio-temporal process in a low-dimensional manifold or parameter reduction (Wikle (2015)), leading to model families quite similar to the ones used in machine learning for modeling dynamical phenomena. Also, for complex underlying processes, observations only provide an incomplete description of the process dynamics so that modeling uncertainty at the data and model levels is an important topic.
In the last 10 years, there has been a growing interest in learning latent representations for example through neural networks and deep learning. Dynamical state space models such as recurrent neural networks (RNN), which have been used for time series forecasting in different contexts since the early nineties (Connor et al. (1994)), have recently witnessed important successes in different areas for general sequence modeling problems, leading to breakthroughs in domains like speech (Graves et al. (2013)), language generation (Sutskever et al. (2011)), translation (Cho et al. (2014)), and many others. Among this family, the model closest to ours is the dynamic factor graph model of (Mirowski & LeCun (2009)) designed for multiple series modeling for the tasks of forecasting and imputation. However this model does not consider relational dependencies which is the focus of our approach.
Most of the above models make use of pointwise representations and do not model explicitly the uncertainties present in the process and/or in the observations. Recently, in the learning representation community, there has been a growing interest in using distributions as latent representations instead of points. (Vilnis & McCallum (2015); He et al. (2015); Dos Santos et al. (2016)) all make use of Gaussian distributions for representing different items like words (Vilnis & McCallum (2015)), nodes in knowledge graphs (He et al. (2015)) or nodes in graphs for transductive classification (Dos Santos et al. (2016)). Note that Gaussian processes have also been used for time series prediction, but they have mainly been considered for univariate time series prediction (Hachino & Kadirkamanathan (2011); Brahim-Belhouari & Bermak (2004)) and they do not use a state space formulation.
Recent techniques in variational inference (Kingma & Welling (2014); Rezende et al. (2014)) deal with uncertainty by modeling distributions in the observation space, mapping random variables within a latent space to observations with a deep neural network. Extension of the variational in-
ference method to time series has been proposed (Fraccaro et al. (2016); Krishnan et al. (2015)) but contrarily to those works, we take into account relationships (both temporal and relational). Furthermore, in our model, we work directly with random variables to predict observations from time series. This gives us direct access to the output distribution with no need to sample or work with intractable distributions.
Our model is built on top of the model in (Ziat et al. (2016)) which proposes a deterministic dynamical process model but does not consider any explicit modeling of uncertainty. In this paper, we propose a model that uses Gaussian embeddings, and extend the dynamics and loss functions of the model in (Ziat et al. (2016)).
3 FORECASTING OF RELATIONAL TIME SERIES
3.1 NOTATIONS AND TASKS
Let us consider a set of n temporal sequences1 x1, ..,xn such that x ptq i P R is the value of the ith sequence at time t defined by xi “ pxp1qi , .., x pT q i q, T being the number of observed time steps. For simplification, we consider that all the series have the same length, but this is not restrictive.
We model the dependencies between the different series through a graph, the different series sources being the graph vertices and the links modeling explicit dependencies between the sources. These links can reflect a spatial proximity between the sources of the series, a similarity of behavior between users or any other predefined relation. These explicit relations will be modeled in the latent space. Our hypothesis is that they will constrain the representation of linked sources to be similar one to another in the latent space, this similarity being controlled by the strength of the link between the two time series, denoted ei,j . We assume that the graph structure is static in time and is provided as a prior information. The model can be extended to learn these static dependencies but this is not considered here.
Let us denote τ the size of the prediction horizon. The forecasting problem considered here is to compute for all series i the values xpT`kqi for all k in r1; τ s. Note that the model can be straightforwardly extended to the imputation problem that aims at predicting missing values.
3.2 INFORMAL DESCRIPTION
The proposed model is a dynamic state space model: the dynamics is modeled in a continuous latent state space and the observations are generated from states in this latent space. State space models have already been considered for multiple time series (e.g. Mirowski & LeCun (2009)) and for spatio-temporal processes (e.g. Wikle & Hooten (2010)).
Both the observations and the dynamics are subject to uncertainties. Usually, the observations correspond to a partial view of the underlying generating process and the dynamics being hidden is not directly accessible and should be modeled as a stochastic process.
To handle this uncertainty, we propose a model, namely Relational Dynamic model with Gaussian representations (RDG), that represents latent factors as distributions in a latent space and learns the series dynamics in this latent space. The distributions themselves are estimated using observations like for any other representation learning model. Besides being more adapted to handling the noise inherent to the process and to the observations, the model can be used to predict the posterior distribution of the variables associated to the series and in particular the confidence or variance associated to the predictions.
The model is an extension of the deterministic model of (Ziat et al. (2016)) and has two main components: (i) Decoding component: we consider that each series corresponds to a particular trajectory in an unknown latent space. Each series xp1qi , ...., x pT q i is thus associated to a series of random variables in Rd denoted Zp1qi , ...., Z pT q i , Z ptq i being the latent factor explaining the observed value of the series i at time t and d the size of the latent space. We model each Zptqi as a multivariate
1For simplicity, we consider univariate time series, but the model can be trivially extended to multivariate time series.
normal variable N pµptqi ,Σ ptq i q. The observation can be computed from this latent distribution by using a decoding function mapping Zptqi to X ptq i “ fpZ ptq i q. (ii) Dynamic component: The second component models the series dynamics in the latent space. We suppose that dynamics can be captured for all series through a function h that maps the latent random variable Zptqi to the next latent variable Zpt`1qi “ hpZ ptq i q. The function h is thus modeling the time dynamics. In addition, constraints are introduced to reflect prior knowledge about the relational dependency structure of the series. For any couple of series i and j with a known dependency, i.e. such that ei,j ą 0 we add a corresponding constraint on Zptqi and Z ptq j as explained in Section 3.3.3.
In the following, we explain how the distributions corresponding to the random variables Zptqi are learned, jointly to the functions f (decoder component) and h (dynamic component).
3.3 MODEL DEFINITION
We suppose that the random variables Zptqi follow a Gaussian distribution. Let us denote Z ptq i „ N pµptqi ,Σ ptq i q a distribution where µ ptq i and Σ ptq i have to be estimated from known observations. For simplicity, we consider in the following that Σptqi is a diagonal matrix, with σ ptq i,j denoting the jth value of the diagonal of Σptqi .
We define a global loss function Lpµ,Σ, f, hqwhere µ and Σ are the means and covariance matrices for all the series and for all the time steps between 1 and T . The loss is a sum of three terms: (i) a decoding loss ∆De, (ii) a dynamical loss ∆Dy and (iii) a structural loss ∆R:
Lpµ,Σ, f, hq “ n ÿ
i“1
T ÿ t“1 ∆DepfpZptqi q, x ptq i q ` λDy n ÿ i“1 T´1 ÿ t“1 ∆DypZpt`1qi , hpZ ptq i qq
` λR n ÿ
j“1
T ÿ t“1 ei,j∆RpZptqi , Z ptq j q (1)
where λDy and λR are hyperparameters weighting the importance of the different elements in the loss function. The first term corresponds to the decoding component, and forces both f and the learned distributions of variables Z to “explain” the observations, the second term, the dynamic component, encourages h to model the time dynamics in the latent space, while the third term captures the relations between the pairs of series. In the following, we use for f a linear function and h will be either a linear or non-linear function (see Section 3.3.2).
Learning: Learning the model is performed through the minimization of the loss function Lpµ,Σ, f, hq with respect to µ, Σ, f and h. To simplify the notations, the parameters of f and h are not made explicit in the notations – f and h are supposed to be differentiable. At the end of the learning process, all the latent distributions for each of the time steps are known for the training data, as well as the decoding function f and the dynamical one h. We used ADAM (Kingma & Ba (2015)) as a stochastic gradient descent technique. This optimization can be easily made on a large scale dataset, and/or by using GPUs.
3.3.1 FROM LATENT SPACE TO OBSERVATIONS
The mapping onto the latent space is learned so that the values xptqi of each series can be predicted from their respective Gaussian embedding Zptqi through the f function. We define below two alternative decoding loss functions ∆De, used in the experiments for measuring the error between the predicted distribution fpZptqi q and the observation x ptq i . Other losses could be used with the same model.
The first loss measures the difference between the expected value of f and the observation using a mean-square error:
∆De1pfpZ ptq i q, x ptq i q def“ ´ E ” fpZptqi q ı ´ xptqi ¯2
(2)
When considering a linear decoding function such as fp¨q “ă θ, ¨ ą , θ being the set of parameters of f , ∆De1 can be rewritten as as:
∆De1pfpZ ptq i q, x ptq i q “ pă θ, µ ptq i ą ´x ptq i q 2 (3)
The second loss aims at measuring the distance between the random variable modeling the predicted observations and the observations. This is the expectation of the mean squared error between the predictions and the observations:
∆De2pfpZ ptq i q, x ptq i q def“ E ” pfpZptqi q ´ x ptq i q 2 ı
(4)
When f is a linear function, this loss can be written as:
∆De2pfpZ ptq i q, x ptq i q “
d ÿ k“1 θ2kσ ptq i,k ` ´ ă θ, µptqi ą ´x ptq i ¯2
(5)
Minimizing ∆De1 only updates the mean of the distributions, whereas minimizing ∆De2 updates both the mean and the variance. More specifically, an observed value with ∆De2 will pull the variances σ ptq i down. This is an interesting property since observing values should reduce the variance of the representation. Moreover, this effect will be higher for the dimensions of the latent space where the value of θ is higher. This is sensible since variance is reduced for the dimensions that are important for the prediction.
3.3.2 MODELING DYNAMICS
The loss function ∆Dy aims at finding values Z p.q i and a dynamic model h, that will be used to predict the representation of the next state of time series i, Zpt`1qi . The function h maps a distribution N pµptqi ,Σ ptq i q to N pµ pt`1q i ,Σ pt`1q i q. Based on (Vilnis & McCallum (2015); Dos Santos et al. (2016)), we use a Kullback-Leibler divergence (noted DKLp¨||¨q) to compare the distribution at pt` 1q to the distribution predicted by h. We propose in the following two alternative functions for h. For the first one, we consider that the latent representation at time pt` 1q is a linear transformation of the latent distribution at time t. The transformed variable is also a Gaussian and its parameters can be easily computed. In this case, h is a linear function from Rd to Rd which is represented by a matrix γ PMd,dpRq:
∆Dy1pZ pt`1q i , hpZ ptq i qq def“ DKLpZpt`1qi ||γZ ptq i q “ DKLpZ pt`1q i ||N pγµ ptq i , γΣ ptq i γ T qq (6)
Linear transformations of random vectors might be too restrictive to model complex processes. As an alternative transformation, we used two non linear multilayer perceptrons (MLP), one hm for predicting the means and one for hc for predicting the variance: the next mean is given by µ pt`1q i “ hmpµ ptq i ,Σ ptq i q, and the next variance by Σ pt`1q i “ hcpµ ptq i ,Σ ptq i q. This gives:
∆Dy2pZ pt`1q i , hpZ ptq i qq def“ DKLpZpt`1qi ||N ph mpµptqi ,Σ ptq i q, h cpµptqi ,Σ ptq i qqq (7)
Note hat in the second case, we also make the hypothesis that the resulting distribution (for Zpt`1qi ) is Gaussian. In the two cases, the KL divergence between the two Gaussian distributions has a simple analytic form from which the gradient can be easily computed2.
3.3.3 STRUCTURAL REGULARIZATION TERM
At last, ∆R corresponds to a structural regularization over the graph structure that encourages the model to learn similar representations for time series that are interdependent. This forces the model to learn representations that reflect the structure dependencies between the series. Recall that these
2DKLpZptqi ||Z ptq j q “ 12 ptrpΣ ptq j ´1 Σ ptq i q ` pµ ptq j ´ µ ptq i qT Σ ptq j ´1 pµptqj ´ µ ptq i q ´ d´ logp |Σptqi |
|Σptqj | qq
dependencies are supposed to be provided as priors for this model. We define this regularization loss as:
∆RpZptqi ||Z ptq j q “ DKLpZ ptq i ||Z ptq j q (8)
which again has, for Gaussian random variables, a simple analytical form that can be used for learning.
Minimizing the regularization term ∆R has a direct impact on the distributions of the predicted observations for connected times series. More precisely, we have the following inequality:
dTV
´
fpZptqi q, fpZ ptq j q
¯
ď
d
d ¨DKLpZptqi ||Z ptq j q
2 (9)
with dTV being the total variation distance of probability measures, i.e.:
dTV pX,Y q “ sup APBorel p|DXpAq ´DY pAq|q (10)
with X and Y being to random variables of density distribution respectively DX and DY , and Borel being the Borel set of Rn (roughly, cuboids in Rn). This means that having relatively similar representations (regarding the KL-divergence) constrains the predicted values to be similar. For more details see Appendix A.
3.4 INFERENCE
During inference when forecasting values, the latent distributions at pT ` 1q are deduced from the ones at time T and follow N phpµpT qi ,Σ pT q i qq, distributions at pT ` 2q follow N ph ˝hpµ pT q i ,Σ pT q i qq, and so on...
4 EXPERIMENTS
4.1 DATASETS AND BASELINES
Experiments have been performed on four datasets respectively extracted from Google Flu Trends3, WHO4 and from two datasets from Grand Lyon5 (GL) (respectively data from traffic conditions and from car parks occupancy). All the series are normalized. For all datasets, we used binary dependency relations indicating whether two series are related or not. The Google Flu Trend (GFT) dataset is composed of an aggregation of weekly Google search queries related to the flu in 29 countries. This dataset spans about ten years of time. The binary relations between series are defined a priori so that the series of two countries i and j are linked, i.e. ei,j “ 1 in Equation (1), only if the countries have a common frontier. There are 96 relations in all. The GL Traffic (GL-T) dataset corresponds to the traffic conditions of the 50 busiest roads of the city of Lyon (France). Data is aggregated on 20 minutes windows spanning 15 days. The binary relations between series are based on the geographical proximity of roads. There are 130 relations in total. The GL Park (GL-P) dataset represents the occupancy of public car parks in Lyon. The series correspond to the occupancy of the 30 busiest car parks. It has the same window and period of time as the previous dataset, and the binary relations between series are based on the geographical proximity of car parks. There are 74 relations in total. The WHO dataset provides the number of deaths caused by diphtheria over 91 different countries, giving rise to 91 time series. The binary relations between series are defined so that two series are linked if the corresponding countries share a common frontier. There are 228 links in total.
We compare our approach with five baselines : Auto-Regressive (AR), a monovariate linear auto-regressive model. It computes its predictions based on a learned linear function of a fixed number p of past values of the series. The order p of the model is a hyperparameter of the model selected by a grid search. Feed Forward Neural Network (FFNN), representative of non-linear
3http://www.google.org/flutrends 4http://www.who.int 5http://data.grandlyon.com
auto-regressive models of order p where the non-linear function is modeled as a feed-forward neural network with one hidden layer of size s. In this case, p and s are hyperparameters selected by grid search. RNN, a recurrent neural network with one hidden layer of size s of recurrent units and tanh non-linearities. The RNN model is a state space non-linear auto-regressive model with exogenous inputs (the past values of the series). Note that this model should in principle be able to learn the inter-series dependencies, but the dependencies are not modeled explicitly as they are in our model. Also the RNN does not introduce explicit modeling of uncertainties. KF (Kalman (1960)), is a classic Kalman Filter with linear transformations from one state to another. DFG (Mirowski & LeCun (2009)), a state of the art model that learns continuous deterministic latent variables by modeling the dynamics and the joint probabilities between series. All the hyperparameters of the baselines have been set using a validation set by grid search, including the best architectures for the dynamic model hwhen it is a multi-layer perceptron with one hidden layer or a linear model. .
For the evaluation we have considered a root-mean-square error (RMSE) criterion. Regarding the experimental protocol, models are evaluated using cross-validation with rolling origin.
4.2 RESULTS
Let us first present the performance of our model w.r.t. the baselines for prediction at horizon 1 in Figure 1b We have tested the four variants of our approach i.e combinations of ∆De1 or ∆De2 with ∆Dy1 or ∆Dy2 . The proposed model obtains the best results on all the datasets except GFT where KF performs better. Otherwise it outperforms the baselines on two datasets (GL-P -Grand Lyon Parksand GFT -Google Flu Trends- on the table) and gets results similar to the RNN on the two others (GL-T -Grand yon Traffic- and WHO). The non linear dynamical model used for ∆Dy2 usually gets better results than other models, the best combination being the use of the MSE expectation error for the decoder and the non-linear model for the dynamics (denoted RDG2,2 on the figure). The non linear dynamical model used for ∆Dy2 usually gets better results than other models, the best combination being the use of the MSE expectation error for the decoder and the non-linear model for the dynamics (denoted RDG2,2 on the figure).
Figure 1a shows the prediction quality (RMSE) at pT `1q, pT `2q, pT `3q, pT `4q and pT `5q and illustrates the ability of RDG to predict correctly at different horizons. Here again, the performance of RDG is very close to the performance of the Recurrent Neural Network. One can remark that at pT ` 5q KF does not goes the distance since it performs well at pT ` 1q but quite badly at pT ` 5q in comparison to other baselines.
RDG has the additional property of modeling the uncertainty associated to its predictions, which is not the case for a RNN. Let us consider the curves presented in Figure 2. They illustrate, the predictions made by our model together with their associated variance computed through the Gaussian embeddings. First, one can see that the ground truth values are always within the confidence interval provided by our model, which means that RDG computes relevant minimum and maximum possible values. Another aspect is that the size of the interval increases with the prediction horizon, which is
what is expected from such a model. The latter is then able to predict relevant confidence values for its predictions.
Comparison between RDG with/without structural regularization or uncertainty We compare in Table 1 the results between our model when taking into account the neighborhood graph (λR ‰ 0) or not (λR “ 0): forecasts are uniformly worse for all datasets when we do not take into account the neighborhood graph, it suggests that the regularizer improves the model when the input graph is relevant. Furthermore, we give the results obtained without uncertainty, which corresponds to the model described in (Ziat et al. (2016)) (denoted Rainstorm): here again, our model outperforms the previous one for all the datasets.
5 CONCLUSION AND FUTURE WORK
We have proposed a model for relational time series forecasting. Our model (RDG) is based on latent Gaussian embeddings, and has shown competitive performance on four different datasets compared to state-of-the-art models. Moreover, RDG allows us to model the uncertainty of predictions, providing for example confidence intervals for each prediction. Future work will investigate more complex dynamic and prediction functions, as well as observing the behavior of the model for imputation tasks.

Because the authors did not respond to reviewer feedback, I am maintaining my original review score.
-----
This paper proposes to model relational (i.e., correlated) time series using a deep learning-inspired latent variable approach: they design a flexible parametric (but not generative) model with Gaussian latent factors and fit it using a rich training objective including terms for reconstruction (of observed time series) error, smoothness in the latent state space (via a KL divergence term encouraging neighbor states to be similarly distributed), and a final regularizer that encourages related time series to have similar latent state trajectories. Relations between trajectories are hard coded based on pre-existing knowledge, i.e., latent state trajectories for neighboring (wind speed) base stations should be similar. The model appears to be fit using gradient simple descent. The authors propose several elaborations, including a nonlinear transition function (based on an MLP) and a reconstruction error term that takes variance into account. However, the model is restricted to using a linear decoder. Experimental results are positive but not convincing.
Strengths:
- The authors target a worthwhile and challenging problem: incorporating the modeling of uncertainty over hidden states with the power of flexible neural net-like models.
- The idea of representing relationships between hidden states using KL divergence between their (distributions over) corresponding hidden states is clever. Combined with the Gaussian distribution over hidden states, the resulting regularization term is simple and differentiable.
- This general approach -- focusing on writing down the problem as a neural network-like loss function -- seems robust and flexible and could be combined with other approaches, including variants of variational autoencoders.
Weaknesses:
- The presentation is a muddled, especially the model definition in Sec. 3.3. The authors introduce four variants of their model with different combinations of decoder (with and without variance term) and linear vs. MLP transition function. It appears that the 2,2 variant is generally better but not on all metrics and often by small margins. This makes drawing a solid conclusions difficult: what each component of the loss contributes, whether and how the nonlinear transition function helps and how much, how in practice the model should be applied, etc. I would suggest two improvements to the manuscript: (1) focus on the main 2,2 variant in Sec. 3.3 (with the hypothesis that it should perform best) and make the simpler variants additional "baselines" described in a paragraph in Sec. 4.1; (2) perform more thorough experiments with larger data sets to make a stronger case for the superiority of this approach.
- The authors only allude to learning (with references to gradient descent and ADAM during model description) in this framework. Inference gets its one subsection but only one sentence that ends in an ellipsis (?).
- It's unclear what is the purpose of introducing the inequality in Eq. 9.
- Experimental results are not convincing: given the size of the data, the differences vs. the RNN and KF baselines is probably not significant, and these aren't particularly strong baselines (especially if it is in fact an RNN and not an LSTM or GRU).
- The position of this paper is unclear with respect to variational autoencoders and related models. Recurrent variants of VAEs (e.g., Krishnan, et al., 2015) seem to achieve most of the same goals as far as uncertainty modeling is concerned. It seems like those could easily be extended to model relationships between time series using the simple regularization strategy used here. Same goes for Johnson, et al., 2016 (mentioned in separate question).
This is a valuable research direction with some intriguing ideas and interesting preliminary results. I would suggest that the authors restructure this manuscript a bit, striving for clarity of model description similar to the papers cited above and providing greater detail about learning and inference. They also need to perform more thorough experiments and present results that tell a clear story about the strengths and weaknesses of this approach.


PERCEIVING FROM LOW FIDELITY VISUAL INPUT
Humans perceive their surroundings in great detail even though most of our visual field is reduced to low-fidelity color-deprived (e.g., dichromatic) input by the retina. In contrast, most deep learning architectures deploy computational resources homogeneously to every part of the visual input. Is such a prodigal deployment of resources necessary? In this paper, we present a framework for investigating the extent to which connectionist architectures can perceive an image in full detail even when presented with low acuity, distorted input. Our goal is to initiate investigations that will be fruitful both for engineering better networks and also for eventually testing hypotheses on the neural mechanisms responsible for our own visual system’s ability to perceive missing information. We find that networks can compensate for low acuity input by learning global feature functions that allow the network to fill in some of the missing details. For example, the networks accurately perceive shape and color in the periphery, even when 75% of the input is achromatic and low resolution. On the other hand, the network is prone to similar mistakes as humans; for example, when presented with a fully grayscale landscape image, it perceives the sky as blue when the sky is actually a red sunset.
1 INTRODUCTION
Most deep learning architectures process every visual input component when performing a task; for example, the input layer of many ImageNet architectures considers all pixels in every region of a preprocessed image when learning an image classifier or making classification decisions (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2014). In contrast, the human visual system has just a small fovea of high resolution chromatic input allowing it to more judiciously budget computational resources (Lennie, 2003). In order to receive additional information in the field of view, we make either covert or overt shifts of attention. Overt shifts of attention or eyemovements allow us to bring the fovea over particular locations in the environment that are relevant to current behavior. To avoid the serial nature of processing as demanded from overt shifts of attention, our visual system can also engage in covert shifts of attention in which the eyes remain fixated on one location but attention is deployed to a different location.
Yet, even without resorting to overt shifts of attention, we still perceive the world in high detail. This is somewhat remarkable if you consider that the human retina receives an estimated 10 million bits per second which far exceeds the computational resources available to our visual system to assimilate at any given time (Koch et al., 2006). Our own fovea takes up only 4% of the entire retina (Michels & CP Rice, 1990) and is solely responsible for sharp central full color vision with maximum acuity; acuity which diminishes rapidly with eccentricity from the fovea (Cowey &
∗Email: fwick@cs.umb.edu
Rolls, 1974). As a result, visual performance is best at the fovea and progressively worse towards the periphery (Low, 1946). Indeed, our visual cortex is receiving distorted color-deprived visual input except for the central two degrees of the visual field (Hansen et al., 2009). Additionally, we have blind spots in the retina that receive no visual input. Yet, we are mostly unaware of these distortions. Even when confronted with actual blurry or distorted visual input, our visual system is good at extracting the scene contents and context. For instance, our system can recognize faces and emotions expressed by those faces in resolutions as low as 16 x 16 pixels (Sinha et al., 2006). We can reliably extract contents of a scene from the gist of an image (Oliva, 2005) even at low resolutions (Potter & Levy, 1969; Judd et al., 2011). Recently, Ullman et al. (Ullman et al., 2016) has shown that our visual system is capable of recognizing contents of images from critical feature configurations (called minimal recognizable images or MIRCs) that current deep learning systems cannot utilize for similar tasks. These MIRCS resemble foveations on an image and their results reveal that the human visual system employs features and processes that are not used by current deep networks. Similarly, little attention has been given to how these networks deal with distorted or noisy inputs. We draw inspiration from the abilities of the human visual system and propose a framework to study questions related to whether an artificial neural network can learn to perceive an image from low fidelity input.
In this paper, we want to understand what kind of information can be gleaned from low-fidelity inputs. What can be gleaned from a single foveal glimpse? What is the most predictive region of an image? We present a framework for studying such questions based on autoencoders. In contrast to traditional or denoising autoencoders (Vincent et al., 2008), which learn to reconstruct the original image in the presence of random salt and pepper noise, our autoencoders attempt to reconstruct original high-detail inputs from systematically corrupted lower-detail foveated versions of those images (that is, images that are entirely low detail except perhaps a small “fovea” of high detail). Thus, we have taken to calling them defoveating autoencoders (DFAE). We find that even relatively simple DFAE architectures are able to perceive color, shape and contrast information, but fail to recover high-frequency information (e.g., textures) when confronted with extremely impoverished input. Interestingly, as the amount of detail present in the input diminishes, the structure of the learnt features becomes increasingly global.
2 RELATED WORK
Corrupting the input or hidden layers via noise to improve neural networks is an area of active study (Bishop, 1995; LeCun et al., 1989; Vincent et al., 2010; Rifai et al., 2011; Xie et al., 2012; Schuler et al., 2013). A highly related framework is denoising autoencoders in which the input (or sometimes hidden layer) is corrupted and the network must reproduce the non-corrupted output (Vincent et al., 2010). However, the form of our input corruption is systematic, not random. Further, the emphasis of our work is to understand to what extent a given architecture can perform perceptual filling-in (Komatsu, 2006), our brain’s ability to perceive content that is not explicitly present in our visual input, from retina-like distorted inputs. We study this capability by varying the type of input distortion and examining (qualitatively and quantitatively) the ability of the network to perceive.
Note that one can consider a low-resolution image as yet another type of noisy input. Thus, another area of related work is image super-resolution in which the goal is to learn a transform from a lowresolution image to a high-resolution image (Behnke, 2001; Cui et al., 2014; Dong et al., 2014), and also image denoising (Jain & Seung, 2009). These are exciting applications for deep learning, but again, our emphasis is on studying a specific set of scientific questions rather than engineering a solution to a specific image-cleaning problem.
There has been tremendous interest in applying attention to deep learning architectures (Larochelle & Hinton, 2010; Mnih et al., 2014; Bahdanau et al., 2014; Xu et al., 2015). Such work has lead to improvements in tasks ranging from machine translation (Bahdanau et al., 2014) to image captioning (Xu et al., 2015). In many of these frameworks, attention interacts with the limited resources in various ways (either by sequentially directing attention (Mnih et al., 2014) or by allowing the network to recall relevant information from the past history (Xu et al., 2015)). Our work is complementary in that we aim to study the limited resource itself: in this work, a single foveal glimpse. We further suggest a way of extending our framework to incorporate attention, but we save investigation for future work.
3 FRAMEWORK: DEFOVEATING AUTOENCODERS (DFAE)
We now present a framework for studying the extent to which neural networks can “perceive” an image given various types of low-detail (or foveated) inputs. We begin by specifying a space of neural network architectures and by precisely defining a notion of perceives that we can measure. It is important that the framework is general and not dependent on a specific task such as image classification in which, for example, the ability to learn domain-specific discriminating features might make it easy to solve the classification problem without fully modeling the structure of the input. This is undesirable because then we are unable to trust classification accuracy as a reliable surrogate for perceiving.
With this in mind, we focus instead on generative models of the raw input data itself, specifically autoencoders (AE). The AE’s hidden units h are analogous to the intermediate neurons in our visual system that capture features and structure of the visual input. Similarly, the AE’s weights W forge visual memories of the training set and are thus analogous to long-term memory. When these weights are properly trained, the activations of the hidden units reflect how the network is perceiving a novel input. However, since these units are not directly interpretable, we indirectly measure how well the network perceives by evaluating the similarity between the original and generated (high-detail) images: the more similar the images are, the better the network is able to perceive.
More formally, let x be the original input image and x̂ = φ(x) be a lower-detail foveated version of that image. That is, a version of the image which is mostly low-detail (e.g., downsampled, black-and-white, or both) except for possibly a small portion which is high-detail (mimicking our own fovea). For example, if we encode images as vectors of floats between 0 and 1 (reflecting pixel intensities in RGB or grayscale) then we might define a class of foveation functions as φ : [0, 1]n → [0, 1]m s.t. m n and the foveation function might downsample the original image according to the eccentricity from the image center while also removing most of the vector components corresponding to color. We then employ the autoencoder to defoveate x̂ by generating a high-quality output image y = f(x̂;W ) in which, for example, y ∈ [0, 1]n. Finally, we can then measure the similarity between y and x as (1) a surrogate for how well the network perceives from the foveated input and (2) part of a loss function to train the network.
In summary, DFAEs simply comprise:
1. A foveation function that filters an original image by removing detail (color, downsampling, blurring, etc). We will later make this the independent variables in our experiments so we can study the effect of different types of input distortion.
2. An autoencoder network that inputs the low-detail foveated input, but is trained to output the high-detail original image.
3. A loss function for measuring the quality of the reconstruction against the original image and for training the network.
Note that much like denoising autoencoders, these autoencoders reconstruct an original image from a corrupted version. However, the form of corruption is a systematic foveation instead of random noise (Vincent et al., 2008). Thus, we have termed these models defoveating autoencoders or DFAEs.
3.1 DFAE ARCHITECTURE AND LOSS FUNCTION
In our experiments, we study DFAEs with fully connected layers. That is, DFAEs of the form: x̂ = φ(x) h(0) = tanh ( W (0)x̂ ) h(i) = tanh ( W (i)h(i−1) ) for i = 1, · · · , k − 1 y = σ ( W (k)h(k−1)
) where σ is the logistic function. The sigmoid in the final layer conveniently allows us to compare the pixel intensities between the generated image y and the original image x directly, without having to post-process the output values. We experiment with the number of hidden units per-layer as well as the number of layers. For training, we could employ the traditional mean-squared (MSE) error or cross-entropy loss, but we found that the domain-specific loss function of peak signal-to-noise ratio
(PSNR) yielded much better training behavior. The PSNR between generated image a y = f(φ(x)) and its original input x is defined as follows:
LH(x, y) = log10
( 1√
MSE(x, y)
) where MSE(x, y) = n−1(x− y)T (x− y) (1)
Network parameters were initialized at random in the range [-0.1,0.1] and loss was minimized by stochastic gradient descent with adagrad updates (Duchi et al., 2011).
3.2 RECURRENT DFAES FOR SEQUENCES OF FOVEATIONS
The above architecture is useful for studying single foveations, which is the primary focus of this work. However, we remark that it is straightforward to augment DFAEs with recurrent connections to handle a sequence of foveations similar to what has has been done for solving classification tasks with attention (Mnih et al., 2014). First, augment the foveation function φ to include a locus ` on which the fovea is centered. Second, a saccade function s(ht;Ws) predicts such a locus from the DFAE’s current hidden states ht, and finally we make the hidden state recurrent via a function g(ht−1,Wg). Putting this all together yields the following architecture:
x̂t = φ(xt, `t) foveate the image at location ` ht = fe(g(ht−1;Wg), x̂t;W ) encode: compute new hidden states `t = s(ht;Ws) compute new locus for next foveation yt = fd(ht) decode: reconstruct high detail image
Now the DFAE can handle a sequence of foveations, making it to possible to study the effects of overt attention and also explore the ability to learn from a sequence of foveations (like a language model) rather than a ground-truth image (the analog of which is not available to a human). However, interactions between attention and input acuity are beyond the scope of this work.
4 EXPERIMENTS
We are interested in the question of whether an artificial neural network can perceive an image from foveated input images and we evaluate this ability by measuring how well the autoencoders can reconstruct a full-resolution image from the low-detail foveated input. In these experiments, we fix the architecture of our network to the family described in the previous section and vary the type of foveation, the number of hidden units and the number of layers and study the learnt features and reconstruction accuracy. We address the following questions: Can the network perceive aspects of the image that are not present in the input? What can it perceive and under what conditions? Can the network perceive color in the periphery? How much capacity is required to perceive missing details? What is the nature of network’s learnt solution to the problem of filling-in?
4.1 FOVEATION FUNCTIONS
In our experiments, we study several different foveation functions. In many cases, downsampling is employed as part of the foveation function for which we employ the nearest neighbor interpolation algorithm. We chose nearest neighbor because it is especially brutal in that it does not smooth over neighboring pixels when interpolating. Foveation functions include:
• downsampled factor d (DS-D): no fovea is present, the entire image is uniformly downsampled by a factor of d using the nearest neighbor interpolation method. For example, a factor of 4 transforms a 28x28 image to a 7x7 image and approximately 94% of the pixels are removed. For color images we downsample each channel (RGB) separately, resulting in color distortion. We test factors of 2, 4, 7 (MNIST) and 2, 4, 8 (CIFAR).
• scotoma r (SCT-R): entire regions (r = 25%, 50% and 75%) of the image are removed (by setting the intensities to 0) to create a blind spot/region, but the rest of the image remains at the original resolution. We experiment with the location of the scotoma (centered or not).
• fovea r (FOV-R): only a small fovea r of high resolution (r = 25% or 6%); the rest of the image is downsampled by a factor of 4.
• achromatic r (ACH-R): only a region of size r has color; color is removed from the periphery by averaging the RGB channels into a single grayscale channel.
• fovea-achromatic r (FOVA-R): combines the fovea r with the achromatic region: only the foveated region is in color, the rest of the image is in grayscale and downsampled by a factor of 4.1
1Note that the achromatic periphery we study here is a more severe distortion than the human periphery, which has dichromatic color reception; though this varies from one individual to another.
4.2 DATASETS AND PRE-PROCESSING
We used two datasets in our experiments: MNIST and CIFAR100. The MNIST database consists of 28 x 28 handwritten digits and has a training set of 60,000 examples and a test set of 10,000 examples. Therefore each class has 6000 examples. The CIFAR100 dataset consists of 32 x 32 color images of 100 classes. Some examples of classes are: flowers, large natural outdoor scenes, insects, people, vehicles etc. Each class has 600 examples. The training set consists of 50,000 images and the test set consists of 10,000 images. We trained DFAEs on the MNIST and CIFAR100 dataset (in grayscale and color). We normalized the datasets so that the pixel values are between 0 and 1 and additionally, zero-centered them. This step corresponds to local brightness and contrast normalization. Aside from this, we do no other preprocessing such as patch extraction or whitening.
4.3 THE EFFECT OF INPUT ACUITY ON THE NETWORK
The purpose of this experiment is to study how various levels of input acuity affect the performance of the network for the case in which no fovea is available. In addition to input acuity (downsampling factors of 1,2,4,7,8), the other variables to consider are the number of hidden units per layer and the number of layers. In pilot experiments we determined that when the number of hidden units was less than the downsampled input size, DFAEs performed poorly; further, that —unlike CNNs for image classification—additional hidden layers (beyond two) did not improve performance much. Therefore, we report results with overcomplete representations of just one or two hidden layers. First, to establish baselines, we compare one of the networks to various interpolation-based upsampling methods available in image editing software (nearest-neighbor, bilinear, bicubic). The single layer DFAE outperforms these standard algorithms on both datasets (see Figure 1).
For qualitative evaluation, Figure 2 contains examples of the reconstructed images by a single layer DFAE. The images produced by the DFAE is compared to upsampled reconstructions via bilinear interpolation. Especially on MNIST, DFAEs can correctly extract the contents of a downsampled input even when 94% of pixels are removed. A compelling example is that even when faced with a blank input (due to aggressive nearest-neighbor downsampling), as seen in Figure 2a, the DFAE can correctly perceive the digit 1. Though in general, the performance of DFAEs suffered when the input was downsampled beyond a factor of 4. When the DFAE made predictions based on the extremely impoverished input, most of the reconstructions were incorrect. Yet, interestingly, the incorrect reconstructions still resembled digits. We hypothesize that this is due to the global feature functions
learnt by the network. Indeed, we observe in Figure 3 that for MNIST, as the downsampling factor increases, the global structure in the features2 also increases: when the input was downsampled by a factor of two (resp. four, seven), it was forced to learn stroke like features (resp. full/partial digits, superimposed digits). A curiously similar result was observed by Vincent et al. Vincent et al. (2010), where their denoising autoencoder learnt global structures when it was trained on randomly noisy inputs.
On the other hand, many filters learnt on CIFAR100 images were not directly interpretable. Though, in some cases the network learnt global features such as a specific color gradients or locally circular blobs which probably enabled it to be better at reconstructing low frequency shape information, color gist, and landscapes particularly well. The reconstructed natural images as seen in Figure 2b show that the DFAE learnt a smoothing and centering function. The DFAEs employed here could predict the shape of objects in the natural images but not the high frequency details (e.g., texture).
4.4 RECONSTRUCTING FOVEATED INPUTS
Until now, we evaluated DFAEs on uniformly downsampled images, but we are especially interested in the case for which some high resolution input is available, more closely resembling the retina. In this section, we evaluate DFAEs on foveated inputs, SCT-R and FOV-R, as described in Section 4.1.
The scotoma allows us to isolate the contribution of the fovea alone (without help from the periphery). Variable-sized areas of region (r = 25%, 50%, 75%, 75% centered) were removed from the original input. The location of removal was chosen randomly from the four quadrants of the input image, except for the condition where 75% of the image around the center was removed. Since
2each feature is the the final output layer weights corresponding to a particular hidden unit
a majority of the input images have a subject of interest, we tested if the central region contained enough information to reconstruct the rest of the image.
The reconstructions in Figure 5a show the DFAE does not perform well when r > 25%. When r = 50%, the DFAE can only reconstruct landscapes, shape and symmetry, demonstrating its ability to extract low frequency information. When r = 75% and 75% centered, the reconstruction process breaks down and the DFAE cannot predict the input beyond the given region of information. The filters learnt under these conditions look similar to the downsampled condition, but with larger blobs.
In FOV-R inputs, r is the same as SCT-R inputs and we chose to use downsampling factor 4 for regions outside the fovea since previous experiments revealed that DFAEs cannot reconstruct inputs downsampled beyond this factor. Figure 5b shows the reconstructed images from FOV-R inputs and Figure 5c show the error rate of reconstruction. The cluster of red lines with lower error rates show that the DFAE performed considerably well with FOV-R than SCT-R inputs. The performance was better ( 1% error for r = 75% centered) than an DFAE trained with uniformly downsampled inputs (1.5% error). This result is not surprising, given that FOV-R contains additional information from regions outside the fovea. These results suggests that a small number of foveations containing rich details might be all these neural networks need to extract contents of the input in higher detail.
4.5 RECONSTRUCTING COLOR FROM FOVEATED INPUTS
It is well known that the human visual system loses chromatic sensitivity towards the periphery of the retina. Recently, there has been interest in how deep networks, specifically convolutional neural networks (CNNs), can learn to color grayscale images (Dahl) and learn artistic style Gatys et al. (2015). Specifically in Dahl reconstructions from grayscale images, numerous cases of the colorized images produced were muted or sepia colored. The problem of colorization which is inherently illposed was treated as a classification task in these studies. Can DFAEs perceive color if it is absent in the input?
We investigated this question using ACH-R and FOVA-R inputs described in section 5.1. The regions of color tested were r = 0% or no color, 6%, 25% and 100% or full color. Figure 6a and 6b show examples of color reconstructions of the these input types. When the DFAE is trained with full color ACH-R inputs, it can make mistakes in reconstructing the right color as seen in Figure 6a. For example: it colors the yellow flower as pink and the purplish-red landscape as blue. When the input is grayscale (no color, r = 0%), the colorizations are gray, muted, sepia toned or simply incorrect in the case of landscapes. But if there is a fovea of color, the single layer DFAE can reconstruct the colorizations correctly. Of course, if the fovea of color is reduced, i.e. 6%, the color reconstruction accuracy falls off but not too drastically. For example, it predicts a yellowish tone for the sunflower among a bed of brown leaves. The critical result is that the performance difference between 100% or full colored inputs and foveated color inputs is small as seen in Figure 6c and 6d. These results suggest that color reconstructions can be just as accurate if these networks can figure out the color of one region of the image accurately as opposed to every region in the image. Similar to the human visual system, these networks are capable of determining accurate colors in the periphery if color information is available at foveation.
5 DISCUSSION AND CONCLUSIONS
We presented a framework for studying whether a given neural network architecture can perceive visual details not explicitly present in the input. Using the framework, we studied a simple fullyconnected network and found it could fill in missing details such as shape, color and contrast, but not texture. We found that the network compensates for missing details by learning global features which when appropriately activated by the surround, effectively infer the missing information. Thus, the network performs perceptual filling-in even though the layers lack the lateral connections and grid-like structure that have been postulated in the isomorphic filling in theory (von der Heydt et al., 2003). Further, the network appears to learn correlations between color and shape: when presented a sunflower input in which only the dark brown center and the inner yellow petals were in color, the network correctly perceived that the surrounding leaves are green; in the case of landscapes, the network incorrectly perceived the sky as blue, even when a small fovea of red sunset is present in the input. Upon inspection, the network indeed learns global color features and these features may
be activated by input patterns corresponding to certain shapes causing the network to occasionally over-generalize, but further investigation is needed to fully confirm this hypothesis.
A potentially interesting line of future research would be to employ the DFAE framework to test the existing hypotheses for how our own visual system performs perceptual filling in: e.g., the process by which our brain infers information that is not explicitly present in our sensory input from the surround (Komatsu, 2006). The specific architectures we studied here are a far cry from the true mechanisms, but in future work we could implement several architectures that capture the essence of competing theories: Which neural architectures exhibit similar behaviors and abilities as humans? Will the architecture be fooled in the same way as humans by optical illusions such as the Cornsweet illusion or the Troxler effect? When does it perceive “The Dress” as blue and gold or white and black? By answering questions such as these, the DFAE framework could provide evidence for one theory (or neural mechanism) over another, and these theories can be further investigated with behavioral and neurological experiments.

This paper aims to characterize the perceptual ability of a neural network under different input conditions.  This is done by manipulating the input image x in various ways (e.g. downsamplig, foveating), and training an auto-encoder to reconstruct the original full-resolution image.  MSE and qualitative results are shown and compared for the different input conditions.
Unfortunately, this paper seems to lack focus, presenting a set of preliminary inspections with few concrete conclusions.  For example, at the end of sec 4.4, "This result is not surprising, given that FOV-R contains additional information .... These results suggests that a small number of foveations containing rich details might be all these neural networks need....".  But this hypothesis is left dangling:  What detailed regions are needed, and from where?  For what sort of tasks?
Secondly, it isn't clear to me what reconstruction behaviors are caused by a fundamental perception of the input, and what are artifacts of the autoencoder and pixelwise l2 loss?  A prime example is texture, which the autoencoder fails to recover.  But with a pixelwise loss, the network must predict high-frequency textures nearly pixel-for-pixel at training time; if this is impossible, then it will generate a pixelwise average of the training samples --- a flat region.  So then the network's inability to reconstruct textures is due to a problem generating them, specifically averaging from the training loss, not necessarily an issue in perceiving textures.  A network trained a different way (perhaps an adversarial network) may infer a texture is there, even if it wouldn't be able to generate it in a pixelwise l2 sense.
Similarly, the ability to perform color reconstruction given a color glimpse I think has much to do with disambiguating the color of an object/scene:  If there is an ambiguity, the network won't know which to "choose" (white flower or yellow flower?) and output an average, which is why there are so many sepia tones.  However, in its section on this, the paper only measures the reconstruction error for different amounts of color given, and does not drill very far into any hypotheses for why this behavior occurs.
There are some interesting measurements here, such as the amount of color needed in the foveation to reconstruct a color image, and the discussion on global features, which may start to get at a mechanism by which glimpses may propagate to an entire reconstruction.  But overall it's hard to know what to take away from this paper.  What are larger concrete conclusions that can be garnered from the details, and what mechanisms bring them about?  Can these be more thoroughly explored with more focus?