Answers.Body: <p>This question has a number of parts to it.</p>
  
  <p>First, you have a <em>representation</em> problem: what is the correct way to present textual data to your machine learning algorithm?</p>
  
  <p>In this case, you chose to apply Bag-of-Words and then TFIDF scores. For English, this might be expected to produce on the order of 100,000 features, with each instance having only a few non-zero features.</p>
  
  <p>If you want to go this route, you would typically also do some kind of <a href="https://en.wikipedia.org/wiki/Feature_selection" rel="nofollow noreferrer">feature selection</a> to eliminate unimportant features from consideration. Depending on your task, you may be able to reduce the size of your input vectors quite dramatically while still getting good performance (for some tasks, to just 100 or so).</p>
  
  <p>You're right that this might not be the most promising approach however.</p>
  
  <p>My choice for this problem would be to use a compression classifier, like <a href="https://plg.uwaterloo.ca/~gvcormac/dmcspam.pdf" rel="nofollow noreferrer">DMC</a>. These have the advantage that they do not need any feature selection or pre-processing, and can easily handle new words or typos. They give state-of-the-art performance on tasks like spam-email classification.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-09T06:05:49.270Z
Answers.Text: Thanks for your advice so far. For your or anyones interest: Doing a test with around 60 000 samples I first got a feature size of around 31 000 from TfIdf (titles only). After applying some normalization to the inputs (e. g. lowercase transformation, removing stopwords, compound splitting) I was able to reduce the features to 13 000. I will have a look at DMC and its possible applications soon.
Answers.UserDisplayName: Mr. Curious
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-08T08:19:14.547Z
Answers.LastActivityDate: 2018-08-08T08:19:14.547Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>This is a huge growth area in the impact of AI on HR -- see all the companies we've found that <a href="https://cognitionx.com/directory/products?f%5Buc%5D%5B0%5D=Candidate%20Matching" rel="nofollow noreferrer">do candidate matching for instance</a> (disclaimer I work for CognitionX). Under the hood, there are techniques that don't rely on vocabulary such as Facebook's FastText but need more training data. </p>
  
  <p>Here are some other resources
  <a href="https://github.com/JAIJANYANI/Automated-Resume-Screening-System" rel="nofollow noreferrer">Job matching using unsupervised learning</a> (k-nearest neighbour)
  <a href="https://i.sstatic.net/1CFA9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1CFA9.png" alt="Excerpt from Oct 2018 paper -- using convolutional neural networks"></a> <a href="https://arxiv.org/abs/1810.04040" rel="nofollow noreferrer">see paper</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-02T20:33:13.013Z
Answers.LastActivityDate: 2019-01-02T20:33:13.013Z
Answers.UserDisplayName: Julian H
Answers.Body: <p>I've been tackling a similar job title classification problem and used this paper as the basis for my approach: <a href="https://web.stanford.edu/~gavish/documents/phrase_based.pdf" rel="nofollow noreferrer">https://web.stanford.edu/~gavish/documents/phrase_based.pdf</a></p>
  
  <p>Might find it useful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T14:58:41.863Z
Answers.LastActivityDate: 2019-07-25T14:58:41.863Z
Answers.UserDisplayName: Jon W
Answers.Body: <p>Its definitely simpler task as NLP or mashine learning, its <strong>keyword</strong> based. And you have a little bit wrong view on that. See your example:</p>
  
  <p><em>Example: java-developer, java web engineer and java software developer shall all be mapped to the profession java engineer.</em>
  Not at all : java web and java are <strong>different</strong> jobs, while java software developer = java-developer and word software means nothing, cause java already stand for software. The info can NOT be mined from texts like job applications -> you have no link to a sence what title is , and better just create the mapping by hand - its not so long. Then, just look in text for key words and ignore other words</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-04T14:19:31.583Z
Answers.Text: _Its definitely simpler task as NLP or mashine learning, its keyword based._ Isn't a keyword based approach just a subpart of NLP? I mean, you will do something like word tokenizing, NER, maybe even POS analysis to improve performance. What's the difference to you?
Answers.UserDisplayName: Mr. Curious
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-01T21:54:03.680Z
Answers.LastActivityDate: 2019-06-01T21:54:03.680Z
Answers.UserDisplayName: user8426627
Body: <p>first of all I want to specify the data available and what needs to be achieved: I have a huge amount of vacancies (in the millions). The information about the <strong>job title</strong> and the <strong>job description</strong> of each vacancy are stored separately. I also have a <strong>list of professions</strong> (around 3000), to which the vacancies shall be mapped.</p>
  
  <p><strong>Example</strong>: <em>java-developer, java web engineer and java software developer</em> shall all be mapped to the profession <em>java engineer</em>.</p>
  
  <p>Now about my current researches and problems: Since a lot of potential training data is present, I thought a machine learning approach could be useful. I have been reading about different algorithms and wanted to give neural networks a shot. </p>
  
  <p>Very fast I faced the problem, that I couldn't find a satisfying way to <strong>transform text of variable length to numerical vectors of constant size</strong> (needed by neural networks). As discussed <a href="https://stackoverflow.com/questions/14783431/processing-strings-of-text-for-neural-network-input">here</a>, this seems to be a non trivial problem. </p>
  
  <p>I dug deeper and came across <a href="https://skymind.ai/wiki/bagofwords-tf-idf" rel="noreferrer">Bag of Words (BOW) and Text Frequency - Inverse Document Frequency (TFIDF)</a>, which seemed suitable at first glance. But here I faced other problems: If I feed all the job titles to TFIDF, the resulting word-weight-vectors will probably be very large (in the tenth of thousands). The search term on the other hand will mostly consist of between 1 and 5 words (we currently match the job title only). Hence, the neural network must be able to reliably map an ultra sparse input vector to one of a few thousand basic jobs. This sounds very difficult for me and I doubt a good classification quality.</p>
  
  <p>Another problem with BOW and TFIDF is, that they cannot handle typos and new words (I guess). They cannot be found in TFIDF's word list, which results in a vector filled with zeros. To sum it up: I was first excited to use TFIDF, but now think it doesn't work well for what I want to do.</p>
  
  <p>Thinking more about it, I now have doubt if neural networks or other machine learning approaches are even good solutions for this task at all. Maybe there are much better algorithms in the field of natural language processing. 
  This moment (before digging into NLP) I decided to first gather the opinions of some more experienced AI users, so I don't miss the best solution. </p>
  
  <p>So <strong>what would be a useful approach to this in your opinion</strong> (best would be an approach that is capable of handling synonyms and typos)? Thanks in advance!</p>
  
  <p>p. s.: I am currently thinking about <strong>feeding the whole job description</strong> into the TFIDF and also do matches for new incoming vacancies with the whole document (instead of job title only). This will expand the size of the word-weight-vector, but it will be less sparse. Does this seem logical to you?</p>
  
Tags: <neural-networks><machine-learning><natural-language-processing>
Title: Is 'job title classification' rather a problem of NLP or machine learning?
UserDisplayName: Mr. Curious
ThreadText: # Is 'job title classification' rather a problem of NLP or machine learning?
  
  (Asked by: username_0 on 2018-07-20)
  
  first of all I want to specify the data available and what needs to be achieved: I have a huge amount of vacancies (in the millions). The information about the **job title** and the **job description** of each vacancy are stored separately. I also have a **list of professions** (around 3000), to which the vacancies shall be mapped.
  
  **Example**: *java-developer, java web engineer and java software developer* shall all be mapped to the profession *java engineer*.
  
  Now about my current researches and problems: Since a lot of potential training data is present, I thought a machine learning approach could be useful. I have been reading about different algorithms and wanted to give neural networks a shot.
  
  Very fast I faced the problem, that I couldn't find a satisfying way to **transform text of variable length to numerical vectors of constant size** (needed by neural networks). As discussed [here](https://stackoverflow.com/questions/14783431/processing-strings-of-text-for-neural-network-input), this seems to be a non trivial problem.
  
  I dug deeper and came across [Bag of Words (BOW) and Text Frequency - Inverse Document Frequency (TFIDF)](https://skymind.ai/wiki/bagofwords-tf-idf), which seemed suitable at first glance. But here I faced other problems: If I feed all the job titles to TFIDF, the resulting word-weight-vectors will probably be very large (in the tenth of thousands). The search term on the other hand will mostly consist of between 1 and 5 words (we currently match the job title only). Hence, the neural network must be able to reliably map an ultra sparse input vector to one of a few thousand basic jobs. This sounds very difficult for me and I doubt a good classification quality.
  
  Another problem with BOW and TFIDF is, that they cannot handle typos and new words (I guess). They cannot be found in TFIDF's word list, which results in a vector filled with zeros. To sum it up: I was first excited to use TFIDF, but now think it doesn't work well for what I want to do.
  
  Thinking more about it, I now have doubt if neural networks or other machine learning approaches are even good solutions for this task at all. Maybe there are much better algorithms in the field of natural language processing.
  This moment (before digging into NLP) I decided to first gather the opinions of some more experienced AI users, so I don't miss the best solution.
  
  So **what would be a useful approach to this in your opinion** (best would be an approach that is capable of handling synonyms and typos)? Thanks in advance!
  
  p. s.: I am currently thinking about **feeding the whole job description** into the TFIDF and also do matches for new incoming vacancies with the whole document (instead of job title only). This will expand the size of the word-weight-vector, but it will be less sparse. Does this seem logical to you?
  
  **Tags:** neural-networks, machine-learning, natural-language-processing
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2018-08-08. Score: 1
  
  This question has a number of parts to it.
  
  First, you have a *representation* problem: what is the correct way to present textual data to your machine learning algorithm?
  
  In this case, you chose to apply Bag-of-Words and then TFIDF scores. For English, this might be expected to produce on the order of 100,000 features, with each instance having only a few non-zero features.
  
  If you want to go this route, you would typically also do some kind of [feature selection](https://en.wikipedia.org/wiki/Feature_selection) to eliminate unimportant features from consideration. Depending on your task, you may be able to reduce the size of your input vectors quite dramatically while still getting good performance (for some tasks, to just 100 or so).
  
  You're right that this might not be the most promising approach however.
  
  My choice for this problem would be to use a compression classifier, like [DMC](https://plg.uwaterloo.ca/~gvcormac/dmcspam.pdf). These have the advantage that they do not need any feature selection or pre-processing, and can easily handle new words or typos. They give state-of-the-art performance on tasks like spam-email classification.
  
  > **Comments:**
  > **username_0**: Thanks for your advice so far. For your or anyones interest: Doing a test with around 60 000 samples I first got a feature size of around 31 000 from TfIdf (titles only). After applying some normalization to the inputs (e. g. lowercase transformation, removing stopwords, compound splitting) I was able to reduce the features to 13 000. I will have a look at DMC and its possible applications soon.  
  
  ---
  
  ### Answer by username_2 on 2019-01-02. Score: 0
  
  This is a huge growth area in the impact of AI on HR -- see all the companies we've found that [do candidate matching for instance](https://cognitionx.com/directory/products?f%5Buc%5D%5B0%5D=Candidate%20Matching) (disclaimer I work for CognitionX). Under the hood, there are techniques that don't rely on vocabulary such as Facebook's FastText but need more training data.
  
  Here are some other resources
  [Job matching using unsupervised learning](https://github.com/JAIJANYANI/Automated-Resume-Screening-System) (k-nearest neighbour)
  [![Excerpt from Oct 2018 paper -- using convolutional neural networks](https://i.sstatic.net/1CFA9.png)](https://i.sstatic.net/1CFA9.png) [see paper](https://arxiv.org/abs/1810.04040)
  
  
  ---
  
  ### Answer by username_3 on 2019-07-25. Score: 0
  
  I've been tackling a similar job title classification problem and used this paper as the basis for my approach: <https://web.stanford.edu/~gavish/documents/phrase_based.pdf>
  
  Might find it useful.
  
  
  ---
  
  ### Answer by username_4 on 2019-06-01. Score: 0
  
  Its definitely simpler task as NLP or mashine learning, its **keyword** based. And you have a little bit wrong view on that. See your example:
  
  *Example: java-developer, java web engineer and java software developer shall all be mapped to the profession java engineer.*
  Not at all : java web and java are **different** jobs, while java software developer = java-developer and word software means nothing, cause java already stand for software. The info can NOT be mined from texts like job applications -> you have no link to a sence what title is , and better just create the mapping by hand - its not so long. Then, just look in text for key words and ignore other words
  
  > **Comments:**
  > **username_0**: _Its definitely simpler task as NLP or mashine learning, its keyword based._ Isn't a keyword based approach just a subpart of NLP? I mean, you will do something like word tokenizing, NER, maybe even POS analysis to improve performance. What's the difference to you?  
  
  ---
  
  