The natural world consists of objects that we perceive as persistent in space and time, even though these objects appear, disappear and reappear in our field of view as we move. This can be attributed to our notion of object persistence -our knowledge that objects typically continue to exist, even if we can no longer see them -and our ability to track objects. Drawing inspiration from the psychology literature on 'sticky indices', we propose the AlignNet, a model that learns to assign unique indices to new objects when they first appear and reassign the index to subsequent instances of that object. By introducing a persistent object-based memory, the AlignNet may be used to keep track of objects across time, even if they disappear and reappear later. We implement the AlignNet as a graph network applied to a bipartite graph, in which the input nodes are objects from two sets that we wish to align. The network is trained to predict the edges which connect two instances of the same object across sets. The model is also capable of identifying when there are no matches and dealing with these cases. We perform experiments to show the model's ability to deal with the appearance, disappearance and reappearance of objects. Additionally, we demonstrate how a persistent object-based memory can help solve question-answering problems in a partially observable environment.
This paper presents a model that is able to associate objects seen in a new time step with the objects seen in previous frames and therefore consider the cases of adding new objects, dropping object no longer visible, but keeping them in memory in case they reappear in the future. The model assumes that the location of the objects is given and the only task to perform is the correct association of instances. Results shows that the proposed approach is better than hand-crafted features on different simulated tasks. Additionally the propose model is shown to help to reduce the computational cost for question answering task.
I lean to reject this paper because in my opinion the proposed method is just a set of predefined rules to train a network to be able to perform object association between a new time step and a memory. Additionally the experimental evaluation is very weak in several points (see below). 
- Contribution: this approach proposes a set of rules to train a network to be able to learn the correct association  between two set of features. Additionally, the network is used together with a memory to keep track of previously seen objects. In my understanding the proposed work presents a set of hard-coded rules to train a network for tracking. However, the connection with tracking is presented only in related works (at the end of the paper) and no comparison with other tracking approach is presented. Additionally, the network model that is actually used for the experiments (graph network) is not presented in the main paper, which makes more difficult to understand how the network works for the given task.
- Experimental evaluation: Experiments are performed on a very simple, simulated environment. Objects are very simple to recognize and their location (which is often  a difficult component of the problem) is given. As the task is quite straight forward, the model obtains almost 100% on all tests. Comparisons are made only with hand-crafted features. It is quite evident that a learned similarity measure will be better than hand-crafted distances. 
Authors should explain more clearly that the proposed contribution is a set of rules used to learn a distance between features in order to associate object instances. 
A cleared connection with tracking should be provided from the introduction. In the evaluation there should be a comparison on multiple and more challenging tracking datasets. In this way we can compare the proposed training technique with other tracking approaches that also learn to a distance in order to associate object at different time steps.