LEARNING REPRESENTATIONS OF SETS THROUGH OPTIMIZED PERMUTATIONS
Representations of sets are challenging to learn because operations on sets should be permutation-invariant. To this end, we propose a Permutation-Optimisation module that learns how to permute a set end-to-end. The permuted set can be further processed to learn a permutation-invariant representation of that set, avoiding a bottleneck in traditional set models. We demonstrate our model's ability to learn permutations and set representations with either explicit or implicit supervision on four datasets, on which we achieve state-of-the-art results: number sorting, image mosaics, classification from image mosaics, and visual question answering.
Update: From the perspective of a "broader ML" audience, I cannot recommend acceptance of this paper. The paper does not provide even a clear and concrete problem statement due to which it is difficult for me to appreciate the results. This is the only paper out of all ICLR2019 papers that I have reviewed / read which has such an issue. Of course for the conference, the area chair / program chairs can choose how to weigh the acceptance decisions between interest to the broader ML audience and the audience in the area of the paper. 
----------------------------------------------------------------------------------------------------------------------------------
 This paper addresses the problem that often features are obtained as a set, whereas certain orders of these features are known to allow for easier learning. With this motivation the goal of this paper is to learn a permutation of the features. This paper makes the following three main contributions:
1. The idea of using pairwise comparison costs instead of position-based costs
2. The methodological crux of how to go from the pairwise comparison costs to the permutation (that is, solving Eqn. (2) using  Eqn. (1) )
3. An empirical evaluation
I like the idea and the empirical evaluations are promising. However, I have a major concern about the second contribution on the method. There is a massive amount of literature on this very problem and a number of algorithms are proposed in the literature. This literature takes various forms including rank aggregation and most popularly the (weighted) minimum feedback arc set problem.  The submitted paper is oblivious to this enormous literature both in the related work section as well as the empirical evaluations. I have listed below a few papers pertaining to various versions of the problem (this list is by no means exhaustive). With this issue, I cannot give a positive evaluation of this submitted paper since it is not clear whether the paper is just re-solving a solved problem. That said, I am happy to reconsider if the related work and the empirical evaluations are augmented with comparisons to the past literature on the methodological crux of the submitted paper (e.g., why off-the-shelf use of previously proposed algorithms may or may not suffice here.)
Unweighted feedback arc set:
A fast and effective heuristic for the feedback arc set problem, Eades et al.
Efficient Computation of Feedback Arc Set at Web-Scale, Simpson et al.
How to rank with few errors, Kenyon-Mathieu et al.
Aggregating Inconsistent Information: Ranking and Clustering, Ailon et al.
Hardness results:
The Minimum Feedback Arc Set Problem is NP-hard for Tournaments, Charbit et al.
Weighted feedback arc set:
A branch-and-bound algorithm to solve the linear ordering problem for weighted tournaments, Charon et al.
Exact and heuristic algorithms for the weighted feedback arc set problem: A special case of the skew‐symmetric quadratic assignment problem, Flood
Approximating Minimum Feedback Sets and Multicuts in Directed Graphs, Even et al.
Random inputs:
Noisy sorting without resampling, Braverman et al.
Stochastically transitive models for pairwise comparisons: Statistical and computational issues, Shah et al.
On estimation in tournaments and graphs under monotonicity constraints, Chatterjee et al. 
Survey (slightly dated):
An updated survey on the linear ordering problem for weighted or unweighted tournaments, Charon et al.
Convex relaxation of permutation matrices:
On convex relaxation of graph isomorphism, Afalo et al.
Facets of the linear ordering polytope, Grotschel