Answers.Body: <p>One of the key terms in the literature that you are looking for is <em>video captioning</em>.</p>
  
  <p>You can have a look at some of the <a href="https://paperswithcode.com/task/video-captioning" rel="nofollow noreferrer">relevant papers with code on this subject</a>. In short, it is an active area of research and a difficult problem, one reason is because videos are still difficult to learn about (because of larger amount of data + larger model, etc...) and this model has to be working with two modalities of data: text and image.</p>
  
  <p>A paper that you might want to start with is <a href="https://cs.stanford.edu/people/karpathy/deepimagesent/" rel="nofollow noreferrer">Deep Visual-Semantic Alignments for Generating Image Descriptions</a> which works on single images. In short, you can use something similar like in the paper: object detector (e.g. Faster RCNN) to extract visual features and feed them into the state of an RNN (LSTM) which would output a sequence of words in your summary (see picture below).
  <a href="https://i.sstatic.net/j28Jx.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/j28Jx.png" alt="image captioning model"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-11T23:31:44.800Z
Answers.Text: Thanks. But how is video captioning related to video scoring? I like to know how a scene is interesting for a viewer.
Answers.UserDisplayName: Mary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-11T23:40:31.787Z
Answers.Text: Right, I focused on "generate say a 10-second short summary given a video as input". Yeah I see your problem. It sounds like a specific problem for which there wouldn't be a dataset available online (don't know at the top of my head). If you have an opportunity to create your own dataset then you can perform regression on your score directly. For example this paper: [YouTube-8M](https://arxiv.org/pdf/1609.08675.pdf) performs video classification. Then instead of predicting a class you would predict a score (also changing the loss to l1 or l2 loss).
Answers.UserDisplayName: Anuar Y
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-12T07:53:32.923Z
Answers.Text: This is a useful answer for a different question, which is why I downvoted it.
Answers.UserDisplayName: DrMcCleod
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-11T23:21:29.513Z
Answers.LastActivityDate: 2019-09-11T23:21:29.513Z
Answers.UserDisplayName: Anuar Y
Body: <p>I am trying to rank video scenes/frames based on how appealing they are for a viewer. Basically, how "interesting" or "attractive" a scene inside a video can be for a viewer. My final goal is to generate say a 10-second short summary given a video as input, such as those seen on Youtube when you hover your mouse on a video.</p>
  
  <p>I previously asked a similar question <a href="https://ai.stackexchange.com/questions/13994">here</a>. But the "aesthetics" model is good for ranking artistic images, not good for frames of videos. So it was failing. I need a score based on "engagement for general audience". Basically, which scenes/frames of video will drive more clicks, likes, and shares when selected as a thumbnail.</p>
  
  <p>Do we have an available deep-learning model or a prototype doing that? A ready-to-use prototype/model that I can test as opposed to a paper that I need to implement myself. Paper is fine as long as the code is open-source. I'm new and can't yet write a code given a paper.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T21:08:14.150Z
Comments.Text: In your model, are you looking for an accurate summary, or want to maximise interest (whilst still limiting output to an edit from the referenced video)? The two goals are often not compatible, witness any film trailer, YouTube "clickbait" etc. I am asking because I think I have seen references to work on the goal of generating accurate summaries, and might be able to find something. But that doesn't appear to be what you want?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T21:14:56.263Z
Comments.Text: Not really accurate summary, but to maximize interest. Yes, it's highly subjective. We don't know the best solution. We just need "a" solution! As long as a model is targeting that concept, it should be fine.
Comments.UserDisplayName: Mary
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T21:16:07.033Z
Comments.Text: @NeilSlater Something like this: https://www.neonopen.org/ they claim their deep models find thumbnails that will drive more clicks, likes, and shares. ​But their codes are client/server based and not easy for me to run.
Comments.UserDisplayName: Mary
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T21:35:44.540Z
Comments.Text: OK, I don't know that area well enough. So I was going to suggest things like https://github.com/Pratik08/vis-dss (although note GPL licensing, which may not suit a commercial product), which is geared around creating an *accurate* summary. It's an interesting area though, so I hope you find an answer. Sometimes accuracy is an attractive end goal of course, if for instance you were indexing video with the goal of helping an end user finding something that they were looking for
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T22:02:25.007Z
Comments.Text: Thanks Neil. I will look into that repo. If anything came into mind within a short term, please let me know.
Comments.UserDisplayName: Mary
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-29T22:21:34.603Z
Comments.Text: @NeilSlater Neil, did you try building the repo yourself?  `cmake` runs ok, but not the `make`.
Comments.UserDisplayName: Mary
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-30T07:08:06.937Z
Comments.Text: No I have not tried building Vis-DSS
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-12T07:52:32.413Z
Comments.Text: This strikes me as an extremely difficult problem due to the subjectivity of the response that you are trying to evaluate. At the very least, you would need to include the demographics of the viewer into the model.
Comments.UserDisplayName: DrMcCleod
Tags: <neural-networks><deep-learning><classification><computer-vision><image-processing>
Title: Video engagement analysis with deep learning
UserDisplayName: Mary
ThreadText: # Video engagement analysis with deep learning
  
  (Asked by: username_0 on 2019-08-27)
  
  I am trying to rank video scenes/frames based on how appealing they are for a viewer. Basically, how "interesting" or "attractive" a scene inside a video can be for a viewer. My final goal is to generate say a 10-second short summary given a video as input, such as those seen on Youtube when you hover your mouse on a video.
  
  I previously asked a similar question [here](https://ai.stackexchange.com/questions/13994). But the "aesthetics" model is good for ranking artistic images, not good for frames of videos. So it was failing. I need a score based on "engagement for general audience". Basically, which scenes/frames of video will drive more clicks, likes, and shares when selected as a thumbnail.
  
  Do we have an available deep-learning model or a prototype doing that? A ready-to-use prototype/model that I can test as opposed to a paper that I need to implement myself. Paper is fine as long as the code is open-source. I'm new and can't yet write a code given a paper.
  
  **Tags:** neural-networks, deep-learning, classification, computer-vision, image-processing
  
  > **Comments:**
  > **username_1**: In your model, are you looking for an accurate summary, or want to maximise interest (whilst still limiting output to an edit from the referenced video)? The two goals are often not compatible, witness any film trailer, YouTube "clickbait" etc. I am asking because I think I have seen references to work on the goal of generating accurate summaries, and might be able to find something. But that doesn't appear to be what you want?  
  > **username_0**: Not really accurate summary, but to maximize interest. Yes, it's highly subjective. We don't know the best solution. We just need "a" solution! As long as a model is targeting that concept, it should be fine.  
  > **username_0**: @username_1 Something like this: https://www.neonopen.org/ they claim their deep models find thumbnails that will drive more clicks, likes, and shares. ​But their codes are client/server based and not easy for me to run.  
  > **username_1**: OK, I don't know that area well enough. So I was going to suggest things like https://github.com/Pratik08/vis-dss (although note GPL licensing, which may not suit a commercial product), which is geared around creating an *accurate* summary. It's an interesting area though, so I hope you find an answer. Sometimes accuracy is an attractive end goal of course, if for instance you were indexing video with the goal of helping an end user finding something that they were looking for  
  > **username_0**: Thanks username_1. I will look into that repo. If anything came into mind within a short term, please let me know.  
  > **username_0**: @username_1 username_1, did you try building the repo yourself?  `cmake` runs ok, but not the `make`.  
  > **username_1**: No I have not tried building Vis-DSS  
  > **username_2**: This strikes me as an extremely difficult problem due to the subjectivity of the response that you are trying to evaluate. At the very least, you would need to include the demographics of the viewer into the model.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-09-11. Score: 1
  
  One of the key terms in the literature that you are looking for is *video captioning*.
  
  You can have a look at some of the [relevant papers with code on this subject](https://paperswithcode.com/task/video-captioning). In short, it is an active area of research and a difficult problem, one reason is because videos are still difficult to learn about (because of larger amount of data + larger model, etc...) and this model has to be working with two modalities of data: text and image.
  
  A paper that you might want to start with is [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://cs.stanford.edu/people/karpathy/deepimagesent/) which works on single images. In short, you can use something similar like in the paper: object detector (e.g. Faster RCNN) to extract visual features and feed them into the state of an RNN (LSTM) which would output a sequence of words in your summary (see picture below).
  [![image captioning model](https://i.sstatic.net/j28Jx.png)](https://i.sstatic.net/j28Jx.png)
  
  > **Comments:**
  > **username_0**: Thanks. But how is video captioning related to video scoring? I like to know how a scene is interesting for a viewer.  
  > **username_3**: Right, I focused on "generate say a 10-second short summary given a video as input". Yeah I see your problem. It sounds like a specific problem for which there wouldn't be a dataset available online (don't know at the top of my head). If you have an opportunity to create your own dataset then you can perform regression on your score directly. For example this paper: [YouTube-8M](https://arxiv.org/pdf/1609.08675.pdf) performs video classification. Then instead of predicting a class you would predict a score (also changing the loss to l1 or l2 loss).  
  > **username_2**: This is a useful answer for a different question, which is why I downvoted it.  
  
  ---
  
  