Answers.Body: <p>The most distinct words of a language are usually the function words (<em>the</em>, <em>and</em>, <em>of</em>, <em>with</em>,...); other lexical items are often (at least partly) shared between languages that had come in contact with each other. So looking for function words is usually the best way to identify the language in a given text.</p>
  
  <p>This can be done by having a list of function words for each language you recognise, look those words up in your text, and then calculate the probability that the text belongs to each language given the frequencies of the words in your list. There will be overlaps, for example Dutch <em>of</em> is equivalent to the conjunction <em>or</em> in English, but also looks like the English <em>of</em>.</p>
  
  <p>Another approach would be to take texts in your languages, and split them into n-grams, typically trigrams. Then you use a list of trigrams and compare the frequency distributions in your known and unknown texts to find which is most likely to match. This has the advantage that you don't need to know anything about the language structure (because you don't need to identify function words), and it also captures the morphology (eg the common English suffix <em>-ing</em>).</p>
  
  <p>You can simply implement this in a fairly basic program, or you could train a machine learning classifier with it. The former seems easier to me, but is not as exciting as the latter.</p>
  
  <p>However, for single words (as per your example) this might not work properly. Generally, the longer the text, the more precise the recognition.</p>
  
  <p>UPDATE: On re-reading your question I think I have misunderstood it, and that you're asking about transliteration into Latin characters. In a way the same applies, that you need to first identify which language it is, and then choose the correct mapping from the Latin transliteration to the writing system that language natively uses.</p>
  
  <p>Since the transliteration is purely symbolic, I don't think that an ML approach would be better than a simple lookup-table of equivalences. And of course there might be difficulties if the writing system has more distinct characters than the  Latin alphabet provides.</p>
  
  <p>UPDATE 2: If the input is transliterated text, then your reference texts (where you know the language) also have to be transliterated in the same character set (eg Latin in your example).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-06T11:19:37.007Z
Answers.Text: @Oliver Mason.: thanks so much for the input. I edited my question. kindly go through it. go here : https://translate.google.co.in/?hl=en&tab=TT&authuser=0 and type "kya", google will say its "Hindi detected". How does that happen?
Answers.UserDisplayName: Hanu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-06T11:19:59.673Z
Answers.Text: @ DuttaA : No characters. just the language is enough. we can easily convert it to hindi characters.
Answers.UserDisplayName: Hanu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-06T08:58:48.960Z
Answers.LastActivityDate: 2019-12-06T11:49:35.987Z
Answers.LastEditDate: 2019-12-06T11:49:35.987Z
Answers.UserDisplayName: Oliver Mason
Body: <p>Assume somebody knows only to write in Latin characters. If they write words of any other language (example: Hindi, French, Latin)  using the Latin alphabet, how can I detect that language?</p>
  
  <p>Example: If they write Hindi language word using the Latin alphabet) </p>
  
  <blockquote>
    <blockquote>
      <p>kya kar raha hai</p>
    </blockquote>
  </blockquote>
  
  <pre><code>     &gt;&gt; the output is Hindi language
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-06T11:48:13.327Z
Comments.Text: You need to distinguish between *alphabet* (Latin, Cyrillic, Greek,  Devangari, ...) and *language* (English, French, Hindi).
Comments.UserDisplayName: Oliver Mason
Tags: <machine-learning><deep-learning><natural-language-processing>
Title: How to detect any native language when written in Latin characters?
UserDisplayName: Hanu
ThreadText: # How to detect any native language when written in Latin characters?
  
  (Asked by: username_0 on 2019-12-06)
  
  Assume somebody knows only to write in Latin characters. If they write words of any other language (example: Hindi, French, Latin)  using the Latin alphabet, how can I detect that language?
  Example: If they write Hindi language word using the Latin alphabet) 
  
  
  kya kar raha hai
  
  
       >> the output is Hindi language
  
  **Tags:** machine-learning, deep-learning, natural-language-processing
  > **Comments:**
  > **username_1**: You need to distinguish between *alphabet* (Latin, Cyrillic, Greek,  Devangari, ...) and *language* (English, French, Hindi).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-06. Score: 2
  
  The most distinct words of a language are usually the function words (*the*, *and*, *of*, *with*,...); other lexical items are often (at least partly) shared between languages that had come in contact with each other. So looking for function words is usually the best way to identify the language in a given text.
  
  This can be done by having a list of function words for each language you recognise, look those words up in your text, and then calculate the probability that the text belongs to each language given the frequencies of the words in your list. There will be overlaps, for example Dutch *of* is equivalent to the conjunction *or* in English, but also looks like the English *of*.
  
  Another approach would be to take texts in your languages, and split them into n-grams, typically trigrams. Then you use a list of trigrams and compare the frequency distributions in your known and unknown texts to find which is most likely to match. This has the advantage that you don't need to know anything about the language structure (because you don't need to identify function words), and it also captures the morphology (eg the common English suffix *-ing*).
  
  You can simply implement this in a fairly basic program, or you could train a machine learning classifier with it. The former seems easier to me, but is not as exciting as the latter.
  
  However, for single words (as per your example) this might not work properly. Generally, the longer the text, the more precise the recognition.
  
  UPDATE: On re-reading your question I think I have misunderstood it, and that you're asking about transliteration into Latin characters. In a way the same applies, that you need to first identify which language it is, and then choose the correct mapping from the Latin transliteration to the writing system that language natively uses.
  
  Since the transliteration is purely symbolic, I don't think that an ML approach would be better than a simple lookup-table of equivalences. And of course there might be difficulties if the writing system has more distinct characters than the Latin alphabet provides.
  
  UPDATE 2: If the input is transliterated text, then your reference texts (where you know the language) also have to be transliterated in the same character set (eg Latin in your example).
  
  > **Comments:**
  > **username_0**: @Oliver Mason.: thanks so much for the input. I edited my question. kindly go through it. go here : https://translate.google.co.in/?hl=en&tab=TT&authuser=0 and type "kya", google will say its "Hindi detected". How does that happen?  
  > **username_0**: @ DuttaA : No characters. just the language is enough. we can easily convert it to hindi characters.  
  
  ---
  
  