<p>As far as I know, this is always done in separate steps. The reason is the availability of training data. For the recognition task (<a href="https://en.wikipedia.org/wiki/Scene_text" rel="nofollow noreferrer">scene text</a> recognition or handwriting recognition), the available data are monolingual.</p>
<p>A solution would be generating synthetic visual data from parallel corpora. Very recently, there was a <a href="https://aclanthology.org/2021.emnlp-main.576" rel="nofollow noreferrer">paper at EMNLP 2021</a> that did exactly this and show high robustness towards OCR errors. However, they do not evaluate the model on real-world images and still assume two steps: recognition and translation.</p>
<p>There is also a domain mismatch between what is typically in handwriting and how the usual data machine translation data look like. From the machine translation perspective, this could be best approached as a domain adaptation problem (e.g, using domain-specific back-translated data). This would quite unhandy with an end-to-end recognition and translation system.</p>

CC BY-SA 4.0
2021-12-17T09:18:15.330Z
2021-12-17T09:18:15.330Z
Jindřich
<p>Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/tokens and then feed those tokens in an RNN)? Is this normally how it would be done, or is there an end-to-end solution?</p>

<convolutional-neural-networks><natural-language-processing><recurrent-neural-networks><machine-translation>
Is image machine translation done in two steps?
cookiecutter
# Is image machine translation done in two steps?

(Asked by: username_0 on 2021-12-11)

Suppose I have images of hand-written Japanese text. If I want to translate those images, would my ML algorithm be a 2-step model (for example, a CNN to convert the image into Japanese characters/tokens and then feed those tokens in an RNN)? Is this normally how it would be done, or is there an end-to-end solution?

**Tags:** convolutional-neural-networks, natural-language-processing, recurrent-neural-networks, machine-translation


---

## 1 answers

### Answer by username_1 on 2021-12-17. Score: 0

As far as I know, this is always done in separate steps. The reason is the availability of training data. For the recognition task ([scene text](https://en.wikipedia.org/wiki/Scene_text) recognition or handwriting recognition), the available data are monolingual.

A solution would be generating synthetic visual data from parallel corpora. Very recently, there was a [paper at EMNLP 2021](https://aclanthology.org/2021.emnlp-main.576) that did exactly this and show high robustness towards OCR errors. However, they do not evaluate the model on real-world images and still assume two steps: recognition and translation.

There is also a domain mismatch between what is typically in handwriting and how the usual data machine translation data look like. From the machine translation perspective, this could be best approached as a domain adaptation problem (e.g, using domain-specific back-translated data). This would quite unhandy with an end-to-end recognition and translation system.


---

