NLP - Hierarchy of words
I am new to natural language processing and I have not heard of a problem similar to mine yet. I was wondering if you could explain me ways of solving this problem or tell me the academic name of this problem so that I can look for the litterature online ?
Here is the problem : From some text (wikipedia articles in a first time), I would like to extract the hierarchy of the different concepts. By hierarchy I mean create a tree where A is a descendant of B if A or one of A's parents (transitive) is defined by B. For instance, normal distribution would be a descendant of probability (since normal distribution is defined using probabilities) and probability would be a descendant (or child) of mathematics. Since it is transitive, normal distribution would also be a child of mathematics.
I had thought of a way of solving this, you can look at the number of times the word A is used alone (called A), the words A and B are used together (called A AND B, 'together' could be for instance in the same article or in the same paragraph, or in the same sentence ...), and the number of times the word B is used alone (called B). Let A be mathematics and B be probability. 330
Then, if the ratios (A AND B)/A and (A AND B)/B are really low then it means there is probably no direct link between A and B (but a link could exist through transitivity). If not, then if A is really bigger than B then probably A is a bigger concept than B. If A and B are almost the same then they are probably siblings (children of the same parent).
Let's take 3 examples:
* Mathematics (A) and carrot (B). A AND B is really low compared to A and B, so there is no direct link between them (or only an indirect link by transitivity).
* Mathematics (A) and probabilities (B). A AND B is quite high compared to B, and A is much bigger than B, so B should be a child of A (probabilities is a child of mathematics).
* Topology (A) and Probabilities (B). A AND B is relativaly high (the texts that present the different areas of mathematics will likely speak about the 2), A and B are about the same order of magnitude, so A and B should be the children of a same parent. Indeed, Topology and Probabilities are the children of Mathematics.
This way of solving the problem is far from perfect, for instance 'the' (A) and 'probability' (B) would probably end up saying probability is a child of the (because A AND B is huge and A is much bigger than B).
I don't know if you know some papers or if you have some ideas to solve this problem ? Also, does my solution seem viable to you and how could we improve it?
Thanks a lot! 882