Answers.Body: <p>Your question is very interesting !</p>
  <p>Actually your example is about multimodal sarcasm detection, one of the last paper on this task (as far I know) is <a href="https://www.internationaljournalssrg.org/IJCSE/paper-details?Id=461" rel="nofollow noreferrer">Detection of Sarcasm through Tone Analysis on video and Audio files: A Comparative Study On Ai Models Performance</a> (The smallest title of the year)
  They create their own dataset using humoristic Youtube video and labelled it manually.
  From the title, it seems to be talking about CV but actually, they are using the first 13 dimensions of MFCC thanks to that they get information about prosody, (pitch and tone) and the shape of the vocal tract, the authors add that information to textual data as input to train their models. (LSTM, CNN and Bi-LSTM)
  Images/videos are not that used yet in Sarcasm identification.</p>
  <p>So, nowadays I think the most achieved combination of NLP and CV is the image text extraction task and its language translation (used in Google Lens for example) which is a combination of Optical Character Recognition and Google Neural Machine Translator.</p>
  <p>If you are looking for a dataset to join CV and NLP recently there is <a href="https://arxiv.org/abs/2106.05606" rel="nofollow noreferrer">VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T10:36:31.407Z
Answers.LastActivityDate: 2022-06-20T11:42:31.130Z
Answers.LastEditDate: 2022-06-20T11:42:31.130Z
Answers.UserDisplayName: Saeron X
Body: <p>I was considering a scenario where natural language processing (NLP) and computer vision (CV) are combined, for example in extended reality systems that get as input both natural language and non-verbal information, e.g. human gestures, and can comprehend it. For example, the agent would get language and non-verbal input and talk to a user.</p>
  <p>How could this be realized? My naive guess would be a conditional transformer, where the conditioning happens on the non-verbal input, but I'm not sure how exactly the conditioning could happen. <strong>What is a current state-of-the-art model for combining NLP and CV?</strong></p>
  <p>Also, are there datasets available for the aforementioned use case? I'm thinking of the scenario where a sentence, e.g.</p>
  <blockquote>
  <p>Yeah, I like him too!</p>
  </blockquote>
  <p>can either mean what is said, and the non-verbal input could be that the person saying it is smiling. However, if the non-verbal input is some laughter, then the above sentence might be meant ironically. <strong>Is there any dataset for this, where sentences and non-verbal inputs are combined?</strong> (Please note that I'm not talking about the generation of a sentence to an image, I'm referring to a <strong>combination</strong> of NLP and CV.)</p>
  <p>Thanks a lot!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-11T21:04:07.780Z
Comments.Text: I don't know the answer, but just in case no one else responds, I can tell you where to look. That's called "multimodal sentiment analysis" in the literature, and the answer you're looking for is probably in this [Google Scholar search](https://scholar.google.com/scholar?as_ylo=2021&q=multimodal+sentiment+analysis&hl=en&as_sdt=7,39).
Comments.UserDisplayName: Lee Reeves
Tags: <natural-language-processing><reference-request><computer-vision>
Title: Current state of the art and datasets for combining NLP and CV?
UserDisplayName: Hermi
ThreadText: # Current state of the art and datasets for combining NLP and CV?
  
  (Asked by: username_0 on 2022-06-11)
  
  I was considering a scenario where natural language processing (NLP) and computer vision (CV) are combined, for example in extended reality systems that get as input both natural language and non-verbal information, e.g. human gestures, and can comprehend it. For example, the agent would get language and non-verbal input and talk to a user.
  
  How could this be realized? My naive guess would be a conditional transformer, where the conditioning happens on the non-verbal input, but I'm not sure how exactly the conditioning could happen. **What is a current state-of-the-art model for combining NLP and CV?**
  
  Also, are there datasets available for the aforementioned use case? I'm thinking of the scenario where a sentence, e.g.
  
  > Yeah, I like him too!
  
  can either mean what is said, and the non-verbal input could be that the person saying it is smiling. However, if the non-verbal input is some laughter, then the above sentence might be meant ironically. **Is there any dataset for this, where sentences and non-verbal inputs are combined?** (Please note that I'm not talking about the generation of a sentence to an image, I'm referring to a **combination** of NLP and CV.)
  
  Thanks a lot!
  
  **Tags:** natural-language-processing, reference-request, computer-vision
  
  > **Comments:**
  > **username_1**: I don't know the answer, but just in case no one else responds, I can tell you where to look. That's called "multimodal sentiment analysis" in the literature, and the answer you're looking for is probably in this [Google Scholar search](https://scholar.google.com/scholar?as_ylo=2021&q=multimodal+sentiment+analysis&hl=en&as_sdt=7,39).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-16. Score: 1
  
  Your question is very interesting !
  
  Actually your example is about multimodal sarcasm detection, one of the last paper on this task (as far I know) is [Detection of Sarcasm through Tone Analysis on video and Audio files: A Comparative Study On Ai Models Performance](https://www.internationaljournalssrg.org/IJCSE/paper-details?Id=461) (The smallest title of the year)
  They create their own dataset using humoristic Youtube video and labelled it manually.
  From the title, it seems to be talking about CV but actually, they are using the first 13 dimensions of MFCC thanks to that they get information about prosody, (pitch and tone) and the shape of the vocal tract, the authors add that information to textual data as input to train their models. (LSTM, CNN and Bi-LSTM)
  Images/videos are not that used yet in Sarcasm identification.
  
  So, nowadays I think the most achieved combination of NLP and CV is the image text extraction task and its language translation (used in Google Lens for example) which is a combination of Optical Character Recognition and Google Neural Machine Translator.
  
  If you are looking for a dataset to join CV and NLP recently there is [VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization](https://arxiv.org/abs/2106.05606)
  
  
  ---
  
  