Answers.Body: <p>As far as I know, this is always done in separate steps. The reason is the availability of training data. For the recognition task (<a href="https://en.wikipedia.org/wiki/Scene_text" rel="nofollow noreferrer">scene text</a> recognition or handwriting recognition), the available data are monolingual.</p>
  <p>A solution would be generating synthetic visual data from parallel corpora. Very recently, there was a <a href="https://aclanthology.org/2021.emnlp-main.576" rel="nofollow noreferrer">paper at EMNLP 2021</a> that did exactly this and show high robustness towards OCR errors. However, they do not evaluate the model on real-world images and still assume two steps: recognition and translation.</p>
  <p>There is also a domain mismatch between what is typically in handwriting and how the usual data machine translation data look like. From the machine translation perspective, this could be best approached as a domain adaptation problem (e.g, using domain-specific back-translated data). This would quite unhandy with an end-to-end recognition and translation system.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-17T09:18:15.330Z
Answers.LastActivityDate: 2021-12-17T09:18:15.330Z
Answers.UserDisplayName: Jindřich
Body: <p>Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/tokens and then feed those tokens in an RNN)? Is this normally how it would be done, or is there an end-to-end solution?</p>
  
Tags: <convolutional-neural-networks><natural-language-processing><recurrent-neural-networks><machine-translation>
Title: Is image machine translation done in two steps?
UserDisplayName: cookiecutter
ThreadText: # Is image machine translation done in two steps?
  
  (Asked by: username_0 on 2021-12-11)
  
  Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/tokens and then feed those tokens in an RNN)? Is this normally how it would be done, or is there an end-to-end solution?
  
  **Tags:** convolutional-neural-networks, natural-language-processing, recurrent-neural-networks, machine-translation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-17. Score: 0
  
  As far as I know, this is always done in separate steps. The reason is the availability of training data. For the recognition task ([scene text](https://en.wikipedia.org/wiki/Scene_text) recognition or handwriting recognition), the available data are monolingual.
  
  A solution would be generating synthetic visual data from parallel corpora. Very recently, there was a [paper at EMNLP 2021](https://aclanthology.org/2021.emnlp-main.576) that did exactly this and show high robustness towards OCR errors. However, they do not evaluate the model on real-world images and still assume two steps: recognition and translation.
  
  There is also a domain mismatch between what is typically in handwriting and how the usual data machine translation data look like. From the machine translation perspective, this could be best approached as a domain adaptation problem (e.g, using domain-specific back-translated data). This would quite unhandy with an end-to-end recognition and translation system.
  
  
  ---
  
  