Relational reasoning-the ability to model interactions and relations between objectsis valuable for robust multi-object tracking and pivotal for trajectory prediction. In this paper we propose MOHART, a class-agnostic, end-to-end multi-object tracking and trajectory prediction algorithm, which explicitly accounts for permutation invariance in its relational reasoning. We explore a number of permutation invariant architectures and show that multi-headed self-attention outperforms the provided baselines and better accounts for complex physical interactions in a challenging toy experiment. We show on three real-world tracking datasets that adding relational reasoning capabilities in this way increases the tracking and trajectory prediction performance, particularly in the presence of ego-motion, occlusions, crowded scenes, and faulty sensor inputs. To the best of our knowledge, MOHART is the first fully end-to-end multi-object tracking from vision approach applied to real-world data reported in the literature.
The paper presents a class-agnostic method for tracking multiple moving objects (MOHART) that extends an existing single-object tracking method (Hierarchical Attentive Recurrent Tracking, HART). Similarly to HART, MOHART utilizes an attention mechanism and LSTM units. The extension form HART to MOHART is done in two main steps: HART is applied to multiple objects in parallel, with a presence variable attached to each, and a permutation-invariant network that learns the interactions between the objects.
The method is tested in two groups of experiments: synthetic data of moving circles, and on several real datasets. The results on the moving circles are enlightening. The paper shows that although MOHART outperforms HART in all circumstances, if the forces between the circles change randomly every number of steps, MOHART outperforms HART by a very large margin. This is because the forces between the circles are determined by their color, which is observable by the network, and MOHART, which capable of learning their interaction, learns the pattern of changing forces. HART, being an independent-particle model, cannot learn the interactions.
The results on the real datasets demonstrate superiority over HART, and in a series of ablation experiments, the Authors are able to show that MOHART's ability to utilize interactions between objects is responsible for the improvement. For example, the improvement over HART is greater when egomotion is significant, because egomotion creates correlations between the movement of objects in the camera frame, which MOHART can utilize.
I found the paper enlightening, based on a neat idea. The experimental results are extensively analyzed and the Author's insights about their algorithm are substantiated by the experiments. 
However the difficulty I had with the paper is that the results are only compared to HART. HART was published in 2017, and the field of scene understanding and tracking had been addressed in multiple papers since then. It is one of the more competitive and application-driven fields in computer vision. Comparing MOHART only to its sister-method from over two years ago significantly limits the usefulness of paper IMHO. The expectation IMHO is not that MOHART outperform each and every method out there, but that the reader know where MOHART stands compared to other methods (e. g. tracking by detection).
I will refrain from pointing to specific papers to compare to, because I believe the Authors should chose the settings, datasets, criteria, and metrics that are the most convenient for them for comparison. The papers surveyed in Section 2, especially under "Tracking-by-Detection", have many followups. Recent papers from those that claim state of the art could good candidates for comparison.
In view of the above, I am inclined at this stage to reject the paper.