Many real-world problems, e.g. object detection, have outputs that are naturally expressed as sets of entities. This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors. We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks. Specifically, in our formulation we incorporate the permutation as unobservable variable and estimate its distribution during the learning process using alternating optimization. We demonstrate the validity of this new formulation on two relevant vision problems: object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO, and a complex CAPTCHA test, where we observe that, surprisingly, our set based network acquired the ability of mimicking arithmetics without any rules being coded.
— Summary
The method extends [21], which proposes an unordered set prediction model for multi-class classification. For that problem, [21] can assume logistic outputs for all distinct classes. This work extends set prediction to the object detection task, where box identity is not distinct — this is handled by an additional model output that reasons about the most likely object permutations. The permutation predictions are used during training, but are not needed at inference time — as shown in Fig1 and Eq 7. Results are on detection of overlapping objects and a CAPTCHA toy summation example.
— Clarity 
The exposition is not particularly clear in several places: 
 - U^m in Eq 1 is undefined and un-discussed. What probability term does it correspond to? It is supposed to make probabilities of different cardinalities comparable, but the exact mechanism is unclear. 
 - The term p(w) disappears on the left hand side of Eq 2. 
 - Notation in Sec. 3.2 is very cumbersome, making it hard to follow. Furthermore, I found the description ambiguous, preventing me from understanding how exactly the permutation head output is used in Eq 5. Specifically, there is some confusion about estimation of w~, which seems based on frequency estimation from past SGD iterations (Eq 3). If so, why does term f2 in Eq 5 contain the permutation head output O2 and how do the two relate? 
 - The network architecture is never described, especially the transition from Conv to Dense and the layer sizes, making the work hard to reproduce. The dimensions of the convolutional feature map matter (probably need to be kept tractable). 
— Significance
Key aspects of the model are not particularly clear, specifically about how the permutation prediction ( the key novelty here) is used to benefit training. 
— Term f2 in Eq5 uses w~ estimates, which appeared to be based on statistics from past SGD runs, yet also depends on the output of the permutation head O2. Am I misinterpreting the method?
— In the paragraph right after Eq5, it’s claimed that “Empirically, in our applications, we found out that estimation of the permutations from just f1 [in Eq5] is sufficient to train properly … by using the Hungarian algorithm”. So then f2 term is not even used in. Eq5? If so, what is the significance of the permutation head other than adding an auxiliary loss? 
Furthermore, there are no experimental results demonstrating the effect of the permutation head and the design choices above — if we could get by with only using the Hungarian algorithm, why bother classifying an exponential number of permutations? Do they help when added as an auxiliary loss?  
While the failure of NMS to detect overlapping objects is expected, the experiments showing that perm-set prediction handles them well is interesting and promising. Solving the general case with larger images and many instances would increase the impact significantly — and likely require a combination of perm-set prediction and image tiling, although this is just a hypothesis. The Captcha toy example also shows some interesting behavior emerging — without digit-specific annotations (otherwise it would be multi-class classification setup from [21]), the model can handle the majority of summations correctly. 
— Experimental results
The results are interesting proofs-of-concept but a few more experiments/answers would be helpful:
- It still appears that PR curve in the high-precision regime (fig 3b) has lower precision than FRCNN/YOLO. Any idea as to why? 
- Ablation results on the effect of the permutation predictions vs Hungarian algorithm, etc would be helpful, as discussed above. 
- How sensitive is the method to seeing a certain cardinality? What if it never sees 3 pedestrians in an image, but only 1,2,4 will it fail to predict 3? Or alternatively, if we train a model that can handle up to 5-6 entities with examples than have <=4? What is the right way of data augmentation for this model (was there any and should there be?)
- Given that values for U differ across applications, how sensitive is the output / how much sweeping did you have to do? 
-- Related work
To the best of my knowledge it's representative. It would help to cite more recent work that decreases detector dependence on NMS. For example, "Learning Non-Maximum Suppression", Hosang, Benenson, Schiele, CVPR 2017 or "Relation Networks for Object Detection", by Hu et al, CVPR 2018 and references therein.