Answers.Body: <p>Huggingface has a <strong><a href="https://huggingface.co/Helsinki-NLP/opus-mt-ka-en" rel="nofollow noreferrer"><code>Helsinki-NLP/opus-mt-ka-en</code></a></strong> repository with a Georgian (Ka) to English (en) model. A <code>tokenizer_config.json</code> is available</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-26T17:42:34.493Z
Answers.Text: i want to build my own translation system. i think it is already trained model for translation, i need tokenizer not whole model. Also i want english to georgian, not vice versa
Answers.UserDisplayName: dato nefaridze
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-26T07:52:57.493Z
Answers.LastActivityDate: 2021-05-26T07:52:57.493Z
Answers.UserDisplayName: Saurav Maheshkar
Body: <p>I want to build a machine translation system from English to Georgian. Georgian is a language similar (and simpler) to the Russian language. its syntax looks like <strong>base</strong> + <strong>suffix</strong>, only <strong>suffix</strong> changes, most of the time <strong>base</strong> is frozen, to describe the time the only suffix is changed. Unfortunately, I couldn't find a morphological analyser for the Georgian language, so could you link or provide useful resources to help me to build one? or can you give me some suggestions?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-26T08:12:32.447Z
Comments.Text: Do you mean a tokeniser or a morphological analyser?
Comments.UserDisplayName: Oliver Mason
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-26T17:47:59.180Z
Comments.Text: i mean tokenizer, i have text( georgian, i will mention english for everybody to understand), "i am dato nefaridze, and living in Georgia", i think it should split these text into words ['i', 'am' 'dato', 'nefaridze' 'and' 'live' 'ing' 'in' 'Georgia']. it should put apart living word
Comments.UserDisplayName: dato nefaridze
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-27T08:46:53.873Z
Comments.Text: That's not a tokeniser. A tokeniser is mostly language-agnostic and only splits a stream of characters into words and non-words. You're looking for a morphological analyser, which can identify component parts of words, and therefore relies on knowledge about the structure of the language.
Comments.UserDisplayName: Oliver Mason
Tags: <natural-language-processing>
Title: How to build a custom morphological analyser for translation system
UserDisplayName: dato nefaridze
ThreadText: # How to build a custom morphological analyser for translation system
  
  (Asked by: username_0 on 2021-05-25)
  
  I want to build a machine translation system from English to Georgian. Georgian is a language similar (and simpler) to the Russian language. its syntax looks like **base** + **suffix**, only **suffix** changes, most of the time **base** is frozen, to describe the time the only suffix is changed. Unfortunately, I couldn't find a morphological analyser for the Georgian language, so could you link or provide useful resources to help me to build one? or can you give me some suggestions?
  
  **Tags:** natural-language-processing
  
  > **Comments:**
  > **username_1**: Do you mean a tokeniser or a morphological analyser?  
  > **username_0**: i mean tokenizer, i have text( georgian, i will mention english for everybody to understand), "i am username_0, and living in Georgia", i think it should split these text into words ['i', 'am' 'username_0', 'username_0' 'and' 'live' 'ing' 'in' 'Georgia']. it should put apart living word  
  > **username_1**: That's not a tokeniser. A tokeniser is mostly language-agnostic and only splits a stream of characters into words and non-words. You're looking for a morphological analyser, which can identify component parts of words, and therefore relies on knowledge about the structure of the language.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-05-26. Score: 1
  
  Huggingface has a **[`Helsinki-NLP/opus-mt-ka-en`](https://huggingface.co/Helsinki-NLP/opus-mt-ka-en)** repository with a Georgian (Ka) to English (en) model. A `tokenizer_config.json` is available
  
  > **Comments:**
  > **username_0**: i want to build my own translation system. i think it is already trained model for translation, i need tokenizer not whole model. Also i want english to georgian, not vice versa  
  
  ---
  
  