We propose a new application of embedding techniques to problem retrieval in adaptive tutoring. The objective is to retrieve problems similar in mathematical concepts. There are two challenges: First, like sentences, problems helpful to tutoring are never exactly the same in terms of the underlying concepts. Instead, good problems mix concepts in innovative ways, while still displaying continuity in their relationships. Second, it is difficult for humans to determine a similarity score consistent across a large enough training set. We propose a hierarchical problem embedding algorithm, called Prob2Vec, that consists of abstraction and embedding steps. Prob2Vec achieves 96.88% accuracy on a problem similarity test, in contrast to 75% from directly applying state-of-the-art sentence embedding methods. It is surprising that Prob2Vec is able to distinguish very fine-grained differences among problems, an ability humans need time and effort to acquire. In addition, the sub-problem of concept labeling with imbalanced training data set is interesting in its own right. It is a multi-label problem suffering from dimensionality explosion, which we propose ways to ameliorate. We propose the novel negative pre-training algorithm that dramatically reduces false negative and positive ratios for classification, using an imbalanced training data set.
This paper proposes a method for mathematical problem embedding, which firstly decomposes problems into concepts by an abstraction step and then trains a skip-gram model to learn concept embedding. A problem can be represented as the average concept (corresponding to those in the problem) embeddings. To handle the imbalanced dataset, a negative pre-training method is proposed to decrease false and false positives. Experimental results show that the proposed method works much better than baselines in similar problem detection, on an undergraduate probability data set. 
Strong points:
(1)	The idea of decomposing problems into concepts is interesting and also makes sense. 
(2)	The training method for imbalanced datasets is impressive. 
Concerns or suggestions:
1.	The main idea of using contents to represent a problem is quite simple and straightforward. The contribution of this paper seems more on the training method for imbalanced data sets. But there are no comparisons between the proposed training method and previous related works. Actually, imbalance data sets are common in machine learning problems and there are many related works. The comparisons are also absent in experiments.
2.	The experimental data set is too small, with only 635 problems. It is difficult to judge the performance of the proposed model based on so small data set. 
3.	The proposed method, which decomposes a problem into multiple concepts, looks general for many problem settings. For example, representing a movie or news article by tags or topics. In this way, the proposed method can be tested in a broader domain and on larger datasets.
4.	For the final purpose, comparing problem similarity, I am wondering what the result will be if we train a supervised model based problem-problem similarity labels?