Answers.Body: <p>Check out <a href="https://arxiv.org/abs/1702.01992" rel="nofollow noreferrer">this paper</a>. It deals with the problem of mixing input modalities! Here's the abstract.</p>
  <blockquote>
  <p>This paper presents a novel model for <strong>multimodal learning</strong> based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for <strong>genre classification of movies using the plot and the poster</strong>. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T02:54:08.800Z
Answers.LastActivityDate: 2020-09-11T15:10:06.043Z
Answers.LastEditDate: 2020-09-11T15:10:06.043Z
Answers.UserDisplayName: Bananin
Body: <p>Are there any good ways of simultaneously incorporating object detection with speech recognition? For example, if you want to identify whether an animal is a dog or cat, we can obviously use visual features (e.g. YOLO, CNNs, etc.). But how would you incorporate speech and sound in this model?</p>
  
Tags: <reference-request><object-detection><speech-recognition>
Title: Are there any good ways of simultaneously incorporating object detection with speech recognition?
UserDisplayName: NebulousReveal
ThreadText: # Are there any good ways of simultaneously incorporating object detection with speech recognition?
  
  (Asked by: username_0 on 2019-11-19)
  
  Are there any good ways of simultaneously incorporating object detection with speech recognition? For example, if you want to identify whether an animal is a dog or cat, we can obviously use visual features (e.g. YOLO, CNNs, etc.). But how would you incorporate speech and sound in this model?
  
  **Tags:** reference-request, object-detection, speech-recognition
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-19. Score: 0
  
  Check out [this paper](https://arxiv.org/abs/1702.01992). It deals with the problem of mixing input modalities! Here's the abstract.
  
  > This paper presents a novel model for **multimodal learning** based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for **genre classification of movies using the plot and the poster**. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.
  
  
  ---
  
  