Deep Interaction Processes for Time-Evolving Graphs
Time-evolving graphs are ubiquitous such as online transactions on an e-commerce platform and user interactions on social networks. While neural approaches have been proposed for graph modeling, most of them focus on static graphs. In this paper we present a principled deep neural approach that models continuous time-evolving graphs at multiple time resolutions based on a temporal point process framework. To model the dependency between latent dynamic representations of each node, we define a mixture of temporal cascades in which a node*s neural representation depends on not only this node*s previous representations but also the previous representations of related nodes that have interacted with this node. We generalize LSTM on this temporal cascade mixture and introduce novel time gates to model time intervals between interactions. Furthermore, we introduce a selection mechanism that gives important nodes large influence in both hop subgraphs of nodes in an interaction. To capture temporal dependency at multiple time-resolutions, we stack our neural representations in several layers and fuse them based on attention. Based on the temporal point process framework, our approach can naturally handle growth (and shrinkage) of graph nodes and interactions, making it inductive. Experimental results on interaction prediction and classification tasks -- including a real-world financial application -- illustrate the effectiveness of the time gate, the selection and attention mechanisms of our approach, as well as its superior performance over the alternative approaches.
The paper is concerned with modeling continuous time-evolving graphs, for which it proposes to combine temporal point processes with a recurrent architecture to learn dynamic node representations. In addition, the paper proposes to stack multiple recurrent layers (to obtain node representations over multiple time scales) and use a temporal attention mechanism (to select relevant past interactions). Modeling temporal and dynamic graphs is an important problem with many applications in ML and AI. The focus of the paper, i.e., to develop improved models by combining TPPs and representation learning, is a promising approach to this task and fits well into ICLR. Furthermore, the presented experimental results are promising. However, I*m concerned about different aspects of the current version: The main contributions of the paper are a recurrent (LSTM-based) architecture to model the intensity function of a TPP, stacking multiple LSTM to form a deep architecture, and a temporal attention mechanism. However, none of these contributions on its own are particularly novel. For instance, prior work that introduces similar approaches include - Recurrent networks to parameterize intensity functions: (Dai, 2017), (Mei & Eisner, 2017), (Trivedi, 2019), ... - Temporal attention: (Trivedi, 2019) Hence, the main novelty seems to lie in the stacked architecture and the particular combination of modules (which is of limited novelty). The experimental results are certainly interesting, but it would be important to provide a more detailed analysis of the model to get insights into the causes for these improvements. With regard to the model: The Log-likelihood function in Section 3.6.1 seems to be incorrect as the LL for a TPP would be L = sum_{i:t_i leq T} loglambda(t_i) - int_0^T lambda(s)ds, which is quite different from the equations in the paper. Is the LL in Section 3.6.1 the actual objective that has been optimized? Regarding the experimental results: All models are trained on the same grid of embedding dimensions, but the proposed method is the only deep model. Hence, its maximum number of parameters can be up to 4x compared to the shallow models. How do the results look if all for models with comparable number of parameters (i.e., can the improvements be explained due to this difference)? It would also be good to get results on commonly used benchmarks (e.g. data used in DyRep or NeuralHawkes) to make the results of the new model comparable to prior experiments and datasets.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deep Interaction Processes for Time-Evolving Graphs
Time-evolving graphs are ubiquitous such as online transactions on an e-commerce platform and user interactions on social networks. While neural approaches have been proposed for graph modeling, most of them focus on static graphs. In this paper we present a principled deep neural approach that models continuous time-evolving graphs at multiple time resolutions based on a temporal point process framework. To model the dependency between latent dynamic representations of each node, we define a mixture of temporal cascades in which a node*s neural representation depends on not only this node*s previous representations but also the previous representations of related nodes that have interacted with this node. We generalize LSTM on this temporal cascade mixture and introduce novel time gates to model time intervals between interactions. Furthermore, we introduce a selection mechanism that gives important nodes large influence in both hop subgraphs of nodes in an interaction. To capture temporal dependency at multiple time-resolutions, we stack our neural representations in several layers and fuse them based on attention. Based on the temporal point process framework, our approach can naturally handle growth (and shrinkage) of graph nodes and interactions, making it inductive. Experimental results on interaction prediction and classification tasks -- including a real-world financial application -- illustrate the effectiveness of the time gate, the selection and attention mechanisms of our approach, as well as its superior performance over the alternative approaches.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper is concerned with modeling continuous time-evolving graphs, for which it proposes to combine temporal point processes with a recurrent architecture to learn dynamic node representations. In addition, the paper proposes to stack multiple recurrent layers (to obtain node representations over multiple time scales) and use a temporal attention mechanism (to select relevant past interactions). Modeling temporal and dynamic graphs is an important problem with many applications in ML and AI. The focus of the paper, i.e., to develop improved models by combining TPPs and representation learning, is a promising approach to this task and fits well into ICLR. Furthermore, the presented experimental results are promising. However, I*m concerned about different aspects of the current version: The main contributions of the paper are a recurrent (LSTM-based) architecture to model the intensity function of a TPP, stacking multiple LSTM to form a deep architecture, and a temporal attention mechanism. However, none of these contributions on its own are particularly novel. For instance, prior work that introduces similar approaches include - Recurrent networks to parameterize intensity functions: (Dai, 2017), (Mei & Eisner, 2017), (Trivedi, 2019), ... - Temporal attention: (Trivedi, 2019) Hence, the main novelty seems to lie in the stacked architecture and the particular combination of modules (which is of limited novelty). The experimental results are certainly interesting, but it would be important to provide a more detailed analysis of the model to get insights into the causes for these improvements. With regard to the model: The Log-likelihood function in Section 3.6.1 seems to be incorrect as the LL for a TPP would be L = sum_{i:t_i leq T} loglambda(t_i) - int_0^T lambda(s)ds, which is quite different from the equations in the paper. Is the LL in Section 3.6.1 the actual objective that has been optimized? Regarding the experimental results: All models are trained on the same grid of embedding dimensions, but the proposed method is the only deep model. Hence, its maximum number of parameters can be up to 4x compared to the shallow models. How do the results look if all for models with comparable number of parameters (i.e., can the improvements be explained due to this difference)? It would also be good to get results on commonly used benchmarks (e.g. data used in DyRep or NeuralHawkes) to make the results of the new model comparable to prior experiments and datasets.
assistant
The paper is concerned with modeling continuous time-evolving graphs, for which it proposes to combine temporal point processes with a recurrent architecture to learn dynamic node representations. In addition, the paper proposes to stack multiple recurrent layers (to obtain node representations over multiple time scales) and use a temporal attention mechanism (to select relevant past interactions). Modeling temporal and dynamic graphs is an important problem with many applications in ML and AI. The focus of the paper, i.e., to develop improved models by combining TPPs and representation learning, is a promising approach to this task and fits well into ICLR. Furthermore, the presented experimental results are promising. However, I*m concerned about different aspects of the current version: The main contributions of the paper are a recurrent (LSTM-based) architecture to model the intensity function of a TPP, stacking multiple LSTM to form a deep architecture, and a temporal attention mechanism. However, none of these contributions on its own are particularly novel. For instance, prior work that introduces similar approaches include - Recurrent networks to parameterize intensity functions: (Dai, 2017), (Mei & Eisner, 2017), (Trivedi, 2019), ... - Temporal attention: (Trivedi, 2019) Hence, the main novelty seems to lie in the stacked architecture and the particular combination of modules (which is of limited novelty). The experimental results are certainly interesting, but it would be important to provide a more detailed analysis of the model to get insights into the causes for these improvements. With regard to the model: The Log-likelihood function in Section 3.6.1 seems to be incorrect as the LL for a TPP would be L = sum_{i:t_i leq T} loglambda(t_i) - int_0^T lambda(s)ds, which is quite different from the equations in the paper. Is the LL in Section 3.6.1 the actual objective that has been optimized? Regarding the experimental results: All models are trained on the same grid of embedding dimensions, but the proposed method is the only deep model. Hence, its maximum number of parameters can be up to 4x compared to the shallow models. How do the results look if all for models with comparable number of parameters (i.e., can the improvements be explained due to this difference)? It would also be good to get results on commonly used benchmarks (e.g. data used in DyRep or NeuralHawkes) to make the results of the new model comparable to prior experiments and datasets.