Numerical common sense (e.g., "a person with a height of 2m is very tall") is essential when deploying artificial intelligence (AI) systems in society. To predict ranges of small and large values for a given target noun and unit, previous studies have implemented a rule-based method that processed numeric values appearing in a natural language by using template matching. To obtain numerical knowledge, crawled textual data from web pages are frequently used as the input in the above method. Although this is an important task, few studies have addressed the availability of numerical common sense extracted from corresponding textual information. To this end, we first used a crowdsourcing service to obtain sufficient data for a subjective agreement on numerical common sense. Second, to examine whether common sense is attributed to current word embedding, we examined the performance of a regressor trained on the obtained data. In comparison with humans, the performance of an automatic relevance determination regression model was good, particularly when the unit was yen (a maximum correlation coefficient of 0.57). Although all the regression approach with word embedding does not predict values with high correlation coefficients, this word-embedding method could potentially contribute to construct numerical common sense for AI deployment.
This paper attempts to study if learned word embeddings for common objects contain information about "numerical common sense". The hypothesis is that certain numerical information may co-occur with the words for certain objects/measurement units within their context windows. To verify this hypotheses, the authors have created a dataset through a crowd-sourcing service which represents "numerical common sense". Using this dataset, the authors examine the predict abilities of regressors trained on learned word embeddings and the aforementioned crowd-sourced dataset. The hypothesis is that if the regressors demonstrate good accuracy, then the word embeddings contained information relevant to "numerical common sense". To the best of my knowledge, this is the first paper that attempts to analyze learned word embeddings in the context of numerical common sense.
This paper should be rejected because (1) the NCS datasets are too small to represent "numerical common sense" (2) the NCS datasets contain faulty data points and (3) the results from the experiments conducted are not sufficient to accept or refute the hypotheses.
Main argument
The first question that we must ask is - are the NCS-50x1 and NCS-60x3 datasets reliable for experiments on "numerical common sense". No, because of two flaws:
(1) The number of samples in the dataset is too small to represent "numerical common sense". Consider the histogram for object "dog" in Figure 2. If the largest data point in this plot was absent, the average of the distribution would be smaller by several orders of magnitude. Perhaps there are other objects in the dataset which are missing samples from the tail end of the distribution that could have large effects on the mean of the collected dataset. 
(2) Some data points in the dataset don't make sense to me. For example, Fig 2 represents the "small" dataset, yet I see samples like 400m long dogs, 40m long cats, 150m long monitors and 20m long mice?
Also, it is not clear how the confidence scores of the participants were taken into account when training the regressors or if they were used at all.
If the NCS dataset does not represent "numerical common sense", it invalidates all experimental results from the paper.
My second issue with the paper is that it is not possible to conclude if the experimental results support or refute the hypothesis (ignoring the issue with the dataset):
1. In tables 2 and 3, the correlation coefficients were quite low and and the MAEs were pretty large. In Table 3, rows 1 and 2, even though the correlation is 0.57 and 0.48, the MAE is 100 million yen and 7.5 million yen respectively which is quite large. To me this suggests that just because the correlation is larger we cannot conclude mean that the model is performing well.
2. It is unclear why the correlation coefficient was chosen to decide that ARD is the superior model in experiment 1. The MAE for random forests with concatenated feature vectors was an order of magnitude smaller than that of the ARD model.
3. Why are the correlation coefficients missing for the unit-only experiment in Table 2? The LS model shows very good MAE relative to the other models and perhaps the correlation should have been measured for that as well? In fact, if the correlation coefficients for this case is comparable to the case with concatenated features, it would mean that the word embedding for the object is not helping at all! Moreover, I find it surprising that the LS model with concatenated features performs worse than the unit-only features. We cannot conclude if paper's interpretation about the results is correct unless this missing information is provided.
4. It is hard to judge what a correlation coefficient of 0.57 means. Why didn't you provide a scatter plot of the predictions vs targets as well? It often happens that even noisy plots demonstrate good correlations.
5. The paper should have additional ablation studies - for example, what would happen in the concatenated feature vector experiment if you trained the regressors using randomly initialized word embeddings instead of the trained word embeddings? Do you get the same performance as learned word embeddings?