Neural Combinatorial Optimization (NCO) has shown the potential to solve traditional NP-hard problems recently. Previous studies have shown that NCO outperforms heuristic algorithms in many combinatorial optimization problems such as the routing problems. However, it is less efficient for more complicated problems such as packing, one type of optimization problem that faces mutual conditioned action space. In this paper, we propose a Conditional Query Learning (CQL) method to handle the packing problem for both 2D and 3D settings. By embedding previous actions as a conditional query to the attention model, we design a fully end-to-end model and train it for 2D and 3D packing via reinforcement learning respectively. Through extensive experiments, the results show that our method could achieve lower bin gap ratio and variance for both 2D and 3D packing. Our model improves 7.2% space utilization ratio compared with genetic algorithm for 3D packing (30 boxes case), and reduces more than 10% bin gap ratio in almost every case compared with extant learning approaches. In addition, our model shows great scalability to packing box number. Furthermore, we provide a general test environment of 2D and 3D packing for learning algorithms. All source code of the model and the test environment is released.
This paper aims at solving geometric bin packing (2D or 3D) problems using a deep reinforcement learning framework. Namely, the framework is based on the actor-critic paradigm, and uses a conditional query learning model for performing composite actions (selections, rotations) in geometric bin packing. Experiments are performed on several instances of 2D-BPP and 3D-BPP,
Overall, bin packing problems are challenging tasks for DRL, and I would encourage the authors to pursue this research topic. Unfortunately, I believe that the current manuscript is at a too early stage for being accepted at ICLR, due to the following reasons:
(a) The paper is littered with spelling/grammar mistakes (just take the second sentence: “With the developing” -> “development”). For the next versions of the manuscript, I would recommend using a spell/grammar checker.
(b) In the related work section, very little is said about Bin Packing Problems. There are various classes of BPPs, and it would be relevant to briefly present them. Moreover, BPPs have been extensively studied in theoretical computer science, with various approximation results. Again, a brief discussion about those results would be relevant. Notably, several classes of geometric bin packing problems admit polynomial-time approximation algorithms (for extended surveys about this topic, see e.g.  Arindam Khan’s Ph.D. thesis 2015; Christensen et. al. Computer Science Review 2017).  
(c) According to the problem formulation and the experiments, it seems that the authors are studying a restricted subclass of 2D/3D bin packing problems: there is only “one” bin, so (it seems that) the authors are dealing with geometric knapsack problems (with rotations). Note that the 2D Knapsack problem with rotations admits a 3/2 + \epsilon - approximation algorithm (Galvez et. al., FOCS 2017). A. Khan has also found approximation algorithms for the 3D Knapsack problem with rotations. So, even if those results do not preclude the use of sophisticated DRL techniques for solving geometric knapsack problems, it would be legitimate to empirically compare these techniques with the polytime asymptotic approximation algorithms already found in the literature.
(d) The problem formulation is very unclear. Namely, the state representation is ambiguous: $s_p$ is obviously not a boolean variable, but a boolean vector (where each component is associated with an item). Nothing is said about actions and transitions and rewards (we have to read the AC framework in order to get a clue of these components). We don’t know if it is an episodic MDP (which is usually the case in DRL approaches to combinatorial optimization tasks). Also, it seems that the MDP is specified for a single instance of 3D-BPP. But this looks wrong since it should include the distribution of all instances of 3D-BPP.   
(e) The Actor-Critic framework, coupled with a conditional query learning algorithm, is unfortunately unintelligible due to the fact that many notations are left unspecified. For example, in Eq (1) what are the dimensions K and V? In Eq (2) what is d_i? In the algorithm what is n_{gae}? Also in the algorithm, what are l’_i, w’_i and h’_i? Etc. 
(f) Even if the aforementioned issues are fixed, it seems that the framework is using many hyper-parameters (\gamma, \beta, \alpha_t, etc.) which are left unspecified. Under such circumstances, it is quite impossible to reproduce experiments.