In-Emulator Machine Translation of Retro Games
It always surprises and annoys me that these two steps—OCR and translation—are put together inseparably in these cloud APIs.<p>OCR is the much simpler of the two, and is possible with an entirely-offline model; but AFAIK there are no “modern” OCR algorithms (e.g. the tech behind WordLens) available as offline libraries. It’s cloud or nothing if you want to get the best algorithms, and <i>especially</i> if you want an OCR algorithm that works on text containing an arbitrary mixture of Unicode graphemes rather than text in a specific known language.<p>And, even if I <i>didn’t</i> just want OCRed text—if you separate the two steps of the process (putting things through OCR first and getting a result back, then feeding those results to machine translation separately), then you also get the opportunity to apply domain-specific “cleaning” to the resultant text before feeding it into machine translation. This usually has much better translation results.<p>One thing I’ve always wanted from web browsers is the ability to just “see into” images, such that I could copy-and-paste text out of any image embedded in a page. (If you’ve ever seen a “translation annotation” done as a floating div on an image, I’d expect to see those, but with text in the original language, not the translated language.) I’ve been consistently surprised when new browsers like Opera or Edge come out and don’t decide to differentiate themselves by including this feature.<p>But, back on topic, I’d love if emulators could “lift” the bitmap text that gets rendered to the framebuffer into semantic “text” visible to the OS accessibility API, in the same way. This would enable:<p>• screen-reader software to read the text<p>• the user to highlight, copy and paste the text<p>• text transcript dumps to be made of the game, using 100% demo-files + scripting<p>Certainly, translation could be added on top of this, but just having the original text, in the original language, <i>available</i> to the emulator’s engine, would open all sorts of possibilities.