We propose a unified framework for building unsupervised representations of entities and their compositions, by viewing each entity as a histogram (or distribution) over its contexts. This enables us to take advantage of optimal transport and construct representations that effectively harness the geometry of the underlying space containing the contexts. Our method captures uncertainty via modelling the entities as distributions and simultaneously provides interpretability with the optimal transport map, hence giving a novel perspective for building rich and powerful feature representations. As a guiding example, we formulate unsupervised representations for text, and demonstrate it on tasks such as sentence similarity and word entailment detection. Empirical results show strong advantages gained through the proposed framework. This approach can potentially be used for any unsupervised or supervised problem (on text or other modalities) with a co-occurrence structure, such as any sequence data. The key tools at the core of this framework are Wasserstein distances and Wasserstein barycenters, hence raising the question from our title.
Pros:
I also study some related tasks and suspect that Wasserstein is helpful for measuring co-occurrence-based similarity. It is nice to see the effort in this direction. 
Cons:
The methods are either not very novel or not very well-motivated. The experiment results are interesting but mixed. If the doubts about the experiments are clarified and the methods are motivated better (or the strengths/weaknesses are better analyzed), I will vote for acceptance.
Related work:
In addition to the work in the related work section, some other work also studied the NLP applications of Wasserstein, especially the ones (such as [1,2,3]) which are related to similarity measurement. The authors should include them in the related work section. 
Question about experiments:
1. Why are the SIF scores reported in Table 1 much lower than the results reported in Arora et al., 2017 and in [4]?
2. If we compare CMD with DIVE + C * delta S, the proposed method wins in EVALution and Weeds, loses in Baroni, Kotlerman, BLESS, and Levy. If you compare DIVE + delta S (Chang et al. 2017) with DIVE + C * delta S, delta S also wins in EVALution and Weeds, loses in Baroni, BLESS, Kotlerman, and Levy (although CMD seems to be better than DIVE + delta S). 
Based on the fact that your method has a high correlation with DIVE + delta S (Chang et al. 2017), I guess that CMD does not work very well when the dataset contains random negative samples, but work well when all the negative samples are similar to the target words. If my guess is right, the performance should be improved on average if you multiply the scores from CMD with the word similarity measurement.
3. To make it efficient, CMD seems to sacrifice some resolutions by using the K representative context. Does this step hurt the performance? Could you provide some performance comparison with different numbers of K to let readers know whether there is a tradeoff between accuracy and efficiency?
4. Since the results are mixed, I suppose readers would like to know when this method will perform better and the reasons for having worse results sometimes.
Writing and presentation suggestions/questions:
1. If the proposed method is a breakthrough, I am fine with the title but I think the experiment results tell us that Wasserstein is not all you need. I understand that everyone wants to have an eye-catching title for their paper. The title of this paper indeed serves this purpose. Since the strategy is effective, more and more people might start to write papers with a title like this. However, having lots of paper called "XXX is all you need?" or "Is XXX all you need?" is definitely not good for the whole community. Please use a more specific title such as Context Mover Distance or something like that.
2. The last point in the contribution is not supported by experiments. I suggest that the authors move this point to the future work section.
3. It is good to see some negative results like Baroni in Table 2. Results on other datasets should not be put into Table 4 in Appendix.
4. Using Wasserstein barycenter to measure sentence similarity seems to be novel, but the motivation is not very clear. Based on A.6, we could see that for each sentence, authors basically find the representative word which is most likely to co-occur with every word in the sentence (has the highest average relatedness rather than similarity) and measure the Wasserstein distance between the co-occurrence probability distribution. I suppose sometimes relatedness is a better metric when measuring sentence similarity, but I think authors should provide some motivative sentence pairs to explain when that is the case.
Using Wasserstein to detect hypernym seems to also be novel, but the motivation is also not clear. Again, a good example would be very helpful.
This point is also related to the last question for experiments.
Minor writing suggestions:
1. In section 3, present the full name of CITE
2. If you put some important equations to the appendix (e.g., the definition of SPPMI_{alpha,gamma}), remember to point readers to the appendix. 
3. In the second paragraph of section 7, Nickel & Kiela, 2017 is a method supervised by a hierarchical structure like WordNet rather than a count-based or word embedding based methods. 
4. In Chang et al., the training dataset is not Wikipedia dump from 2015. This difference of evaluation setup should be mentioned somewhere (e.g., in the caption of Table 2).
5. The reference section is not very organized. For example, the first name of Benotto is missing for the PhD thesis "Distributional Models for Semantic Relations: A Study on Hyponymy and Antonymy". The arXiv papers are cited using different formats. Only some papers have URL. The venue's names are sometimes not capitalized. Gaussian embedding is cited twice, etc.
[1] Kusner, M., Sun, Y., Kolkin, N., & Weinberger, K. (2015). From word embeddings to document distances. In International Conference on Machine Learning (pp. 957-966).
[2] Xu, H., Wang, W., Liu, W., & Carin, L. (2018). Distilled Wasserstein Learning for Word Embedding and Topic Modeling. NIPS 
[3] Rolet, A., Cuturi, M., & Peyré, G. (2016, May). Fast dictionary learning with a smoothed wasserstein loss. In Artificial Intelligence and Statistics (pp. 630-638).
[4] Perone, C. S., Silveira, R., & Paula, T. S. (2018). Evaluation of sentence embeddings in downstream and linguistic probing tasks. arXiv preprint arXiv:1806.06259.