In our previous post we have described the general process of training an N-Gram language detector. If you have not yet read the post describing the general technique I suggest that you have a look at that first. In this post we will go through the implementation details and creation of a simple language detector using Node. To keep this post concise I will set the language detection parameters - top n-grams to select and n-gram length - manually. In a future post I will show you how to optimise the selection of these parameters by splitting the dataset into three parts; training, validation and test.
One of the most important parts of an N-Gram language detector is the creation of language profiles. For this we need some sample text in each language we want to detect. There are many datasets out there which can be used for this purpose, but for this post we will use the Universal Declaration of Human Rights Database since i) it has been translated into over 460 different languages and dialects and ii) it’s free.
One question you might have is whether the source dataset choice affects the quality of our detector. If you remember well from the previous post, the N-Gram distribution for the top 300 or so N-Grams are always highly correlated to the language. It is only when we start making use of more N-Grams that our source document choice will become important since it will start skewing the NGram distribution and make it topical.
After downloading and installing all the dependencies, let us create the function generateProfile which will create an NGram profile. Since the language profiles and document profile are generated using the same process we will extract this common functionality in the file ngram-utils.js.
// Make sure that there is letter a period.
//Discard all punctuation except for Apostrophe.
The getNgrams function takes in an input string text together with the nGram length and returns a list of nGrams frequencies.
The sortNgrams function takes in a dictionary of nGrams and sorts them by frequency storing the index.
The generateProfile combines the functionality provided by these two methods in order to merge a list of bi-grams and tri-grams returning the top N result based on frequency.
Finally, let us create a language profile for each language present in the subset folder and save all profiles in the file language-profile.json. We will make use of this profile file in the next section.
Now that we have created our language profiles, how can we use these profiles to detect the language of a text fragment?
What we need to do is to use the generateProfile function to create a document profile and use this document profile to calculate a simple rank-order statistic which we will call the “out of place” measure.
Now let us read the language profiles we have created in the previous step.
Next, let us read the user text fragment and generate the document profile.
Now it’s time to compute the “out of place” measure for each NGram present in the document profile.
// We found the ngram so we compute the out of place measure.
// We did not find the ngram so we penalise.
Finally, let us sort the score results and present them to the user. Note that the lower the score the “closer” the text fragment is to the language.
As an example, running the detector with the phrase “What is the weather today?” will detect that the phrase is in English.
npm run detect "What is the weather today?"
npm run detect "X'inhu t-temp illum?"
In this post we have implemented an NGram language detector using Node. Hope you guys had fun reading this post! Stay safe and keep hacking!