Answers.Body: <p>It’s seems like quite challenging problem; at least you would need quite a lot of annotated data and computational power.</p>
  
  <p>The approaches/optimizations you could consider:</p>
  
  <ul>
  <li>To make scene change detection and take short piece out of each</li>
  <li>To introduce some kind of “novelty” metric and try to maximize
  it to get most different parts of video   </li>
  <li>To convert video to kind of vector with existing solution like
  r-cnn and yolo and then process it with recurrent networks.   </li>
  <li>The task seems to be very close to video capturing/summarization, you
  can take inspiration there  </li>
  <li>Also, the attention approach might be handy, look, for example
  <a href="https://arxiv.org/abs/1906.02792" rel="nofollow noreferrer">self-attention for video</a>,
  <a href="https://pdfs.semanticscholar.org/5dc7/c33475b545271d1de726fd88bb68dfb7e11b.pdf" rel="nofollow noreferrer">semantic attention for
  video</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T15:24:58.770Z
Answers.Text: Thanks. Yeah first one I'm doing already. Basically all conventional non-DL based approaches. Can you elaborate more on r-cnn and how to use YOLO? The latter is only for object detection, so not sure how.
Answers.UserDisplayName: Mary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-26T08:45:34.333Z
Answers.Text: R-cnn solves the same task as yolo, just another approach.
  
  The problem with video is that they are very high dimensional and you would need a really big dataset to train directly. For example
  - Mnist is 28x28 and you need 50’000 to get result
  - Imagenet 224x224x3 and you need 14’000’000 images
  - 5 minutes video would be 640x480x3x7200. So you would need billions of annotated videos to train directly.
Answers.UserDisplayName: Kirill Fedyanin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-26T08:45:43.140Z
Answers.Text: It’s basically impossible, so you need to reduce the dimensiality in some way. With yolo instead of images you can construst the vector that represents that “we have two objects that look like humans standing near object looking like car”, so, the dimensions would be like 2000x7200 and it would be more managable for DL approaches.
  
  
  Another way would be to understand how you on your own ranking the scenese and why; then you can try manual feature engineering. Here I would argue yolo would be helpful as well, because it’s easier to deal with some semantic data then raw images.
Answers.UserDisplayName: Kirill Fedyanin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-26T20:07:17.343Z
Answers.Text: I see. One idea came to my mind is to tag those shots I'm interested as `1` and others as `0`, and train a model on only those shots against the rest. How do you think it will work? And what's the process?
Answers.UserDisplayName: Mary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T11:17:42.413Z
Answers.LastActivityDate: 2019-07-25T11:17:42.413Z
Answers.UserDisplayName: Kirill Fedyanin
Body: <p>I'm new to machine learning and especially, deep learning. Given a video (and it's subtitle), I need to generate a 10-second summary out of this video. How can I use ML and DL to produce the most representative summary out of this video? More specifically, given video scenes, what are some ways to select and rank them, and how to do it? Any ideas would be helpful.</p>
  
Tags: <neural-networks><machine-learning><deep-learning><convolutional-neural-networks><problem-solving>
Title: Use deep learning to rank video scenes
UserDisplayName: Mary
ThreadText: # Use deep learning to rank video scenes
  
  (Asked by: username_0 on 2019-07-25)
  
  I'm new to machine learning and especially, deep learning. Given a video (and it's subtitle), I need to generate a 10-second summary out of this video. How can I use ML and DL to produce the most representative summary out of this video? More specifically, given video scenes, what are some ways to select and rank them, and how to do it? Any ideas would be helpful.
  
  **Tags:** neural-networks, machine-learning, deep-learning, convolutional-neural-networks, problem-solving
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-25. Score: 1
  
  It’s seems like quite challenging problem; at least you would need quite a lot of annotated data and computational power.
  
  The approaches/optimizations you could consider:
  
  * To make scene change detection and take short piece out of each
  * To introduce some kind of “novelty” metric and try to maximize
    it to get most different parts of video
  * To convert video to kind of vector with existing solution like
    r-cnn and yolo and then process it with recurrent networks.
  * The task seems to be very close to video capturing/summarization, you
    can take inspiration there
  * Also, the attention approach might be handy, look, for example
    [self-attention for video](https://arxiv.org/abs/1906.02792),
    [semantic attention for
    video](https://pdfs.semanticscholar.org/5dc7/c33475b545271d1de726fd88bb68dfb7e11b.pdf)
  
  > **Comments:**
  > **username_0**: Thanks. Yeah first one I'm doing already. Basically all conventional non-DL based approaches. Can you elaborate more on r-cnn and how to use YOLO? The latter is only for object detection, so not sure how.  
  > **username_1**: R-cnn solves the same task as yolo, just another approach.  The problem with video is that they are very high dimensional and you would need a really big dataset to train directly. For example - Mnist is 28x28 and you need 50’000 to get result - Imagenet 224x224x3 and you need 14’000’000 images - 5 minutes video would be 640x480x3x7200. So you would need billions of annotated videos to train directly.  
  > **username_1**: It’s basically impossible, so you need to reduce the dimensiality in some way. With yolo instead of images you can construst the vector that represents that “we have two objects that look like humans standing near object looking like car”, so, the dimensions would be like 2000x7200 and it would be more managable for DL approaches.   Another way would be to understand how you on your own ranking the scenese and why; then you can try manual feature engineering. Here I would argue yolo would be helpful as well, because it’s easier to deal with some semantic data then raw images.  
  > **username_0**: I see. One idea came to my mind is to tag those shots I'm interested as `1` and others as `0`, and train a model on only those shots against the rest. How do you think it will work? And what's the process?  
  
  ---
  
  