As the pace of deep learning innovation accelerates, it becomes increasingly important to organize the space of problems by relative difficultly. Looking to other fields for inspiration, we see analogies to the Chomsky Hierarchy in computational linguistics and time and space complexity in theoretical computer science. As a complement to prior theoretical work on the data and computational requirements of learning, this paper presents an empirical approach. We introduce a methodology for measuring validation error scaling with data and model size and test tasks in natural language, vision, and speech domains. We find that power-law validation error scaling exists across a breadth of factors and that model size scales sublinearly with data size, suggesting that simple learning theoretic models offer insights into the scaling behavior of realistic deep learning settings, and providing a new perspective on how to organize the space of problems. We measure the power-law exponent-the "steepness" of the learning curve-and propose using this metric to sort problems by degree of difficulty. There is no data like more data, but some tasks are more effective at taking advantage of more data. Those that are more effective are easier on the proposed scale. Using this approach, we can observe that studied tasks in speech and vision domains scale faster than those in the natural language domain, offering insight into the observation that progress in these areas has proceeded more rapidly than in natural language.
The paper proposes an empirical solution to coming up with a hierarchy of deep learning tasks or in general machine learning tasks. They propose a two-way analysis where power-law relations are assumed between (a) validation loss and training set size, and (b) the number of parameters of the best model and training set size. The first power-law exponent, \beta_g, indicates how much can more training data be helpful for a given task and is used for ordering the hardness of problems. The second power-law exponent, \beta_p, indicates how effectively does the model use extra parameters with increasing training set (can also be thought of as how good the model is at compression). From experiments across a range of domains, the authors find that indeed on tasks where much of the progress has been made tend to be ones with smaller \beta_g (and \beta_p). It's arguable as to how comparable these power-law exponents are across domains because of differences in losses and other factors, but it's definitely a good heuristic to start working in this direction.   
Clarifications needed:
(a) Why was full training data never used? The plots/analysis would have looked more complete if the whole training data was used, wondering why certain thresholds for data fraction were chosen.
(b) What exactly does dividing the training set into independent shards mean? Are training sets of different sizes created by random sampling without replacement from the whole training set?
(c) How exactly is the "Trend" line fitted? From the right sub-figure in Figure 1, it seems that fitting a straight line in only the power-law region makes sense. But that would require determining the start and end points of the power-law region. So some clarification on how exactly is this curve fitting done? For the record, I'm satisfied with the curve fitting done in the plots but just need the details.
Major issues: 
(a) Very difficult to know in Figure 3 (right) what s(m)'s are associated with which curve, except the one for RHNs maybe.
(b) Section 4.1: In the discussion around Figure 2, I found some numbers a little off. Firstly, using the left plot, I would say that at even 8 images the model starts doing better than random instead of <25 that's stated in the text.  Secondly, the 99.9% classification error rate for top-5 is wrong, it's 99.5% for top-5 ("99.9% classification error rate for top-1 and top-5").
(c) Section 5: The authors use the phrase "low dimensional natural language data" which is quite debatable, to say the least. The number of possible sentences of K length with vocabulary |V| scale exponentially |V|^K where |V| is easily in 10K's most of the time. So to say that this is low dimensional is plain wrong. Just think about what is the (input, output) space of machine translation compared to image classification.
Typos/Suggestions:
(a) Section 3: "Depending the task" -> "Depending on the task"
(b) Section 4.3: "repeatably" -> "repeatability"
(c) Figure 4: Specify number of params in millions. The plot also seems oddly big compared to other plots. Also, proper case the axis labels, like other plots. 
(d) Section 4.2 and 4.2.1 can be merged because the character LM experiments are not discussed in the main text or at least not clearly enough for me.  The values of \beta_g seem to include the results of character LM experiments. So either mention the character LM experiments in more detail or just point to results being in appendix.