Answers.Body: <p>Do you have access to parallel corpora in source and target language that translates idioms correctly? Neural machine translation.  (NMT) should handle this. NMT uses deep learning to match sequences/pairs of words in one language to another and is now the state of the art method for translation AI.</p>
  
  <p>I don't think an AI knows grammar of a language. A translating AI knows patterns but not necessarily grammar in the sense that we learn in school as children. Here's a potential approach that should work give a large enough corpora with examples of idioms - <a href="https://github.com/facebookresearch/MUSE" rel="nofollow noreferrer">github.com/facebookresearch/MUSE</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-05T15:25:47.803Z
Answers.LastActivityDate: 2019-09-05T15:32:12.407Z
Answers.LastEditDate: 2019-09-05T15:32:12.407Z
Answers.UserDisplayName: Adnan S
Answers.Body: <p><strong>Short answer</strong>: Yes.</p>
  
  <p><strong>TL;DR</strong></p>
  
  <p>In the presence of good datasets this can be accomplished with a <a href="https://www.aclweb.org/anthology/E17-3027" rel="nofollow noreferrer">pipeline</a>.</p>
  
  <p><strong>Long Answer</strong></p>
  
  <p>In reality an idiom is a series of words which is supposed to have a semantic meaning that is not denoted by the literal reading (<a href="https://pdfs.semanticscholar.org/6a4e/8ebd0979ae38d693528fe4da30f67db82768.pdf" rel="nofollow noreferrer">source</a>). This means that any system that is used must be capable of considering multiple words at a time. Additionally, some idioms are <strong>context dependent</strong>. Example:</p>
  
  <ul>
  <li>The fisherman broke the ice with his tool.</li>
  </ul>
  
  <p>Are we to believe that this is a very suave fisherman?</p>
  
  <blockquote>
    <p>So, it is possible to teach an AI to use idiomatic phrases to <strong>keep up with the culture of humans</strong>?</p>
  </blockquote>
  
  <p>Observe that humans do not come linguistically "pre-loaded" with idioms. So we can safely assume that idiom usage is a learning task and that the only way for them to keep up is for them to keep learning. So if we solve the idiom learning task we just need to keep our agent <a href="https://en.wikipedia.org/wiki/Online_machine_learning" rel="nofollow noreferrer">online</a> or periodically retrain it on nascent corpora.</p>
  
  <p>One difficulty is that, in the absence of a label, a metaphor could be easily mistaken for an idiom and vice versa. So <a href="https://link.springer.com/chapter/10.1007/978-3-642-37247-6_35" rel="nofollow noreferrer">semantic outlier</a> (sorry it's not free) approaches  may suffer from precision issues. Example:</p>
  
  <ul>
  <li>She's a thorny wildflower (metaphor - could easily be an idiom)</li>
  <li>She's a diamond in the rough (idiom - could easily be a metaphor)</li>
  </ul>
  
  <p>Though, idioms will most likely be repeated if a dataset is large whereas a "custom metaphor" is less likely to repeat. </p>
  
  <p>Additionally, some idioms (eg bite the bullet or break a leg) do not have readily available "interpretable semantics" that allow us to extract their intended meaning. For example, if one did not know the idiom "cut me some slack" one could think:</p>
  
  <p>"Slack implies loosening or to make less tight/taut. I was being very uptight. They probably want me to loosen up and not be so critical."</p>
  
  <p>Of course the human understanding of it might happen in a flash and not follow such a delineated path. The idea is that some <a href="https://www.quora.com/What-is-natural-language-processing-pipeline" rel="nofollow noreferrer">NLP pipeline</a> might be constructible that satisfactorily handles idioms in some specific use cases (example of a <a href="https://www.aclweb.org/anthology/E17-3027" rel="nofollow noreferrer">pipeline</a>). For example, one module might attempt to process outliers like "diamond in the rough" which have said interpretable semantics. Though, something like "bite the bullet" may have to be labelled with the correct semantics.</p>
  
  <p>I've only scratched the surface of this. Natural language understanding is already a hard problem - and idioms are thus a tough task in a tough task. I hope that this motivates the reading of some more thorough articles. I have gathered some articles that can be used as a springboard into the literature. </p>
  
  <p>Here's a <a href="https://nlp.stanford.edu/~muzny/docs/mz-emnlp2013.pdf" rel="nofollow noreferrer">source</a> that uses a dictionary type approach to train the model to recognize idioms. Excerpt:</p>
  
  <blockquote>
    <p>For identification, we assume data of the form <span class="math-container">${(⟨p_i,d_i⟩,y_i) : i = 1...n}$</span> where <span class="math-container">$p_i$</span> is the phrase associated with definition <span class="math-container">$d_i$</span> and <span class="math-container">$y_i ∈ \{literal, idiomatic\}$</span>.</p>
  </blockquote>
  
  <p>This <a href="https://www.aclweb.org/anthology/R15-1087" rel="nofollow noreferrer">source</a> provides pseudo-code for idiom extraction.</p>
  
  <p>This <a href="https://pdfs.semanticscholar.org/6a4e/8ebd0979ae38d693528fe4da30f67db82768.pdf" rel="nofollow noreferrer">source</a> describes a dataset to help solve the idiom difficulties.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-06T15:37:52.403Z
Answers.Text: _Natural language processing is already a hard problem_, to be more precise, I would say natural language _understanding_ is a hard problem.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-06T15:47:50.940Z
Answers.Text: @nbro Indeed. Thanks for the input. I want my answers to be as precise as possible. see edit
Answers.UserDisplayName: respectful
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-05T15:51:29.383Z
Answers.LastActivityDate: 2019-09-06T15:48:19.237Z
Answers.LastEditDate: 2019-09-06T15:48:19.237Z
Answers.UserDisplayName: respectful
Body: <p>After an AI goes through the process described in <a href="https://ai.stackexchange.com/questions/1970/how-would-an-ai-learn-language">How would an AI learn language?</a>, an AI knows the grammar of a language through the process of grammar induction. They can speak the language, but they have learned <em>formal grammar</em>. But most conversations today, even formal ones, use idiomatic phrases. Would it be possible for an AI to be given a set of idioms, for example, </p>
  
  <blockquote>
    <p>Immer mit der Ruhe</p>
  </blockquote>
  
  <p>Which, in German, means 'take it easy' but an AI of grammar induction, if told to translate 'take it easy' to German, would not think of this. And if asked to translate this, it would output</p>
  
  <blockquote>
    <p>Always with the quiet</p>
  </blockquote>
  
  <p>So, it is possible to teach an AI to use idiomatic phrases to keep up with the culture of humans?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-04T03:11:26.213Z
Comments.Text: Do you have access to parallel corpora in source and target language that translates idioms correctly? Neural machine translation should handle this. I don't an AI knows grammar of a language - a translation AI knows patterns but not necessarily grammar in the sense that we learn in school as children. Here's a potential approach that should work give a large enough corpora  with examples of idioms - https://github.com/facebookresearch/MUSE
Comments.UserDisplayName: Adnan S
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-04T23:42:52.473Z
Comments.Text: Can you make this into an answer?
Comments.UserDisplayName: Marvin
Tags: <machine-learning><natural-language-processing>
Title: How would an AI learn idiomatic phrases in a natural language?
UserDisplayName: Marvin
ThreadText: # How would an AI learn idiomatic phrases in a natural language?
  
  (Asked by: username_0 on 2019-09-01)
  
  After an AI goes through the process described in [How would an AI learn language?](https://ai.stackexchange.com/questions/1970/how-would-an-ai-learn-language), an AI knows the grammar of a language through the process of grammar induction. They can speak the language, but they have learned *formal grammar*. But most conversations today, even formal ones, use idiomatic phrases. Would it be possible for an AI to be given a set of idioms, for example,
  
  > Immer mit der Ruhe
  
  Which, in German, means 'take it easy' but an AI of grammar induction, if told to translate 'take it easy' to German, would not think of this. And if asked to translate this, it would output
  
  > Always with the quiet
  
  So, it is possible to teach an AI to use idiomatic phrases to keep up with the culture of humans?
  
  **Tags:** machine-learning, natural-language-processing
  > **Comments:**
  > **username_1**: Do you have access to parallel corpora in source and target language that translates idioms correctly? Neural machine translation should handle this. I don't an AI knows grammar of a language - a translation AI knows patterns but not necessarily grammar in the sense that we learn in school as children. Here's a potential approach that should work give a large enough corpora  with examples of idioms - https://github.com/facebookresearch/MUSE  
  > **username_0**: Can you make this into an answer?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-09-05. Score: 8 | Accepted answer
  
  **Short answer**: Yes.
  
  **TL;DR**
  
  In the presence of good datasets this can be accomplished with a [pipeline](https://www.aclweb.org/anthology/E17-3027).
  
  **Long Answer**
  
  In reality an idiom is a series of words which is supposed to have a semantic meaning that is not denoted by the literal reading ([source](https://pdfs.semanticscholar.org/6a4e/8ebd0979ae38d693528fe4da30f67db82768.pdf)). This means that any system that is used must be capable of considering multiple words at a time. Additionally, some idioms are **context dependent**. Example:
  
  * The fisherman broke the ice with his tool.
  
  Are we to believe that this is a very suave fisherman?
  
  > So, it is possible to teach an AI to use idiomatic phrases to **keep up with the culture of humans**?
  
  Observe that humans do not come linguistically "pre-loaded" with idioms. So we can safely assume that idiom usage is a learning task and that the only way for them to keep up is for them to keep learning. So if we solve the idiom learning task we just need to keep our agent [online](https://en.wikipedia.org/wiki/Online_machine_learning) or periodically retrain it on nascent corpora.
  
  One difficulty is that, in the absence of a label, a metaphor could be easily mistaken for an idiom and vice versa. So [semantic outlier](https://link.springer.com/chapter/10.1007/978-3-642-37247-6_35) (sorry it's not free) approaches may suffer from precision issues. Example:
  
  * She's a thorny wildflower (metaphor - could easily be an idiom)
  * She's a diamond in the rough (idiom - could easily be a metaphor)
  
  Though, idioms will most likely be repeated if a dataset is large whereas a "custom metaphor" is less likely to repeat.
  
  Additionally, some idioms (eg bite the bullet or break a leg) do not have readily available "interpretable semantics" that allow us to extract their intended meaning. For example, if one did not know the idiom "cut me some slack" one could think:
  
  "Slack implies loosening or to make less tight/taut. I was being very uptight. They probably want me to loosen up and not be so critical."
  
  Of course the human understanding of it might happen in a flash and not follow such a delineated path. The idea is that some [NLP pipeline](https://www.quora.com/What-is-natural-language-processing-pipeline) might be constructible that satisfactorily handles idioms in some specific use cases (example of a [pipeline](https://www.aclweb.org/anthology/E17-3027)). For example, one module might attempt to process outliers like "diamond in the rough" which have said interpretable semantics. Though, something like "bite the bullet" may have to be labelled with the correct semantics.
  
  I've only scratched the surface of this. Natural language understanding is already a hard problem - and idioms are thus a tough task in a tough task. I hope that this motivates the reading of some more thorough articles. I have gathered some articles that can be used as a springboard into the literature.
  
  Here's a [source](https://nlp.stanford.edu/~muzny/docs/mz-emnlp2013.pdf) that uses a dictionary type approach to train the model to recognize idioms. Excerpt:
  
  > For identification, we assume data of the form ${(⟨p\_i,d\_i⟩,y\_i) : i = 1...n}$ where $p\_i$ is the phrase associated with definition $d\_i$ and $y\_i ∈ \{literal, idiomatic\}$.
  
  This [source](https://www.aclweb.org/anthology/R15-1087) provides pseudo-code for idiom extraction.
  
  This [source](https://pdfs.semanticscholar.org/6a4e/8ebd0979ae38d693528fe4da30f67db82768.pdf) describes a dataset to help solve the idiom difficulties.
  
  > **Comments:**
  > **username_3**: _Natural language processing is already a hard problem_, to be more precise, I would say natural language _understanding_ is a hard problem.  
  > **username_2**: @nbro Indeed. Thanks for the input. I want my answers to be as precise as possible. see edit  
  
  ---
  
  ### Answer by username_1 on 2019-09-05. Score: 3
  
  Do you have access to parallel corpora in source and target language that translates idioms correctly? Neural machine translation. (NMT) should handle this. NMT uses deep learning to match sequences/pairs of words in one language to another and is now the state of the art method for translation AI.
  
  I don't think an AI knows grammar of a language. A translating AI knows patterns but not necessarily grammar in the sense that we learn in school as children. Here's a potential approach that should work give a large enough corpora with examples of idioms - [github.com/facebookresearch/MUSE](https://github.com/facebookresearch/MUSE)
  
  
  ---
  
  