A HIERARCHICAL MODEL FOR DEVICE PLACEMENT
We introduce a hierarchical model for efficient placement of computational graphs onto hardware devices, especially in heterogeneous environments with a mixture of CPUs, GPUs, and other computational devices. Our method learns to assign graph operations to groups and to allocate those groups to available devices. The grouping and device allocations are learned jointly. The proposed method is trained with policy gradient and requires no human intervention. Experiments with widely-used computer vision and natural language models show that our algorithm can find optimized, non-trivial placements for TensorFlow computational graphs with over 80,000 operations. In addition, our approach outperforms placements by human experts as well as a previous state-of-the-art placement method based on deep reinforcement learning. Our method achieves runtime reductions of up to 60.6% per training step when applied to models such as Neural Machine Translation.
The paper seems clear enough and original enough. The idea of jointly forming groups of operations to colocate and figure out placement on devices seems to hold merit. Where the paper falls short is motivating the problem setting. Traditionally, for determining optimal execution plans, one may resort to cost-based optimization (e.g., database management systems). This paper's introduction provides precisely 1 statement to suggest that may not work for deep learning. Here's the relevant phrase: "the cost function is typically non-stationary due to the interactions between multiple devices". Unfortunately, this statement raises more questions than it answers. Why are the cost functions non-stationary? What exactly makes them dynamic? Are we talking about a multi-tenancy setting where multiple processes execute on the same device? Unlikely, because GPUs are involved. Without a proper motivation, its difficult to appreciate the methods devised.
Pros:
- Jointly optimizing forming of groups and placing these seems to have merit
- Experiments show improvements over placement by human "experts"
- Targets an important problem
Cons:
- Related work seems inadequately referenced. There exist other linear/tensor algebra engines/systems that perform such optimization including placing operations on devices in a distributed setting. This paper should at least cite those papers and qualitatively compare against those approaches. Here's one reference (others should be easy to find): "SystemML's Optimizer: Plan Generation for Large-Scale Machine Learning Programs" by Boehm et al, IEEE Data Engineering Bulletin, 2014.
- The methods are not well motivated. There are many approaches to devising optimal execution plans, e.g., rule-based, cost-based, learning-based. In particular, what makes cost-based optimization inapplicable? Also, please provide some reasoning behind your hypothesis which seems to be that while costs may be dynamic, optimally forming groups and placing them is learn-able.
- The template seems off. I don't see the usual two lines under the title ("Anonymous authors", "Paper under double-blind review").
- The title seems misleading. ".... Device Placement" seems to suggest that one is placing devices when in fact, the operators are being placed.