Answers.Body: <p>What you might want to look for is called <strong>video captioning</strong>. Earlier examples from this line of research is:</p>
  
  <ul>
  <li><p>Venugopalan et. al. <a href="https://arxiv.org/pdf/1505.00487.pdf" rel="nofollow noreferrer">Sequence to Sequence – Video to Text</a>, in ICCV 2015</p></li>
  <li><p>Pasunuru et.al. <a href="http://aclweb.org/anthology/P/P17/P17-1117.pdf" rel="nofollow noreferrer">Multi-Task Video Captioning with Video and
  Entailment Generation</a>, in ACL 2017</p></li>
  </ul>
  
  <p>Below is a screenshot of the results (positive and negative) reported in those papers:
  <a href="https://i.sstatic.net/neTfw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/neTfw.png" alt="Example from Venugopalan1"></a>
  <a href="https://i.sstatic.net/qO1XT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qO1XT.png" alt="Example from Pasunuru"></a></p>
  
  <p>For the ICCV paper, it's not hard to find some implementations, e.g. <a href="https://github.com/vijayvee/video-captioning" rel="nofollow noreferrer">here</a>.</p>
  
  <p>For more recent results, I would suggest to look into the <a href="http://activity-net.org/challenges/2017/captioning.html" rel="nofollow noreferrer">ActivityNet 2017 Challenge - dense captioning</a> or its <a href="http://activity-net.org/challenges/2018/tasks/anet_captioning.html" rel="nofollow noreferrer">2018 version</a>. Some winning solutions include:</p>
  
  <ul>
  <li><a href="https://arxiv.org/pdf/1804.08274.pdf" rel="nofollow noreferrer">Jointly Localizing and Describing Events for Dense Video Captioning</a></li>
  <li><a href="https://arxiv.org/pdf/1806.08854.pdf" rel="nofollow noreferrer">RUC+CMU: System Report for Dense Captioning Events in Videos</a></li>
  </ul>
  
  <p>However I am not sure whether any open-source implementation has been released.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-01T03:43:37.717Z
Answers.LastActivityDate: 2018-09-01T03:43:37.717Z
Answers.UserDisplayName: user12075
Answers.Body: <p>Here is one open-source implementation. <a href="https://github.com/dot27/temporal-tessellation" rel="nofollow noreferrer">Temporal Tessellation: A Unified Approach for Video Analysis</a></p>
  
  <p>For more you can dig into some research publications and see if they give a link to their implementation. Most researchers put up their work for public.</p>
  
  <p>Here is a list of publications that present their work related to video captioning using machine learning. <a href="https://github.com/kjw0612/awesome-deep-vision#video-captioning" rel="nofollow noreferrer">Awesome Deep Vision</a></p>
  
  <p>Here is another publication that shows how to generate captions for videos. <a href="https://www.ics.uci.edu/~yyang8/research/video-caption/video-caption.pdf" rel="nofollow noreferrer">Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-03T09:58:22.547Z
Answers.LastActivityDate: 2018-03-03T10:04:36.093Z
Answers.LastEditDate: 2018-03-03T10:04:36.093Z
Answers.UserDisplayName: Mohsin
Answers.Body: <p>No , even generating brief summary of a video is out of scope as of current state of the art. Training such a model is a tough task. Video understanding is pretty far from the research as of now. 
  But you could try generating descriptions of some keyframes of the video , and align them to form meaningful passages.</p>
  
  <p>check this <a href="https://arxiv.org/abs/1611.06607" rel="nofollow noreferrer">https://arxiv.org/abs/1611.06607</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-01T13:23:44.763Z
Answers.LastActivityDate: 2018-07-01T13:23:44.763Z
Answers.UserDisplayName: thecomplexitytheorist
Body: <p>I'd like to generate subtitles for a silent film. Is there an open source project out there capable of creating captions based on a series of images (such as a scene from a movie)?</p>
  
  <p>EDIT: thanks for the comments below. To clarify, what i'm looking for is an algorithm which can generate a caption for a sequences of images within a movie  describing what happens in the sequence. This is for preliminary research, so accuracy is less important. </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-28T18:56:15.767Z
Comments.Text: This is a really cool question.  My guess, based on current limitations, is that there is no AI in existence that could come close to providing this function... today.  But even as AI advances, this particular function will be challenging.  The AI has to understand the context AND subtext of the silent scenes.  *(Many humans seem not to get subtext;)*  Both are far beyond current capabilities, but I'm sure facial recognition related to emotions will be a major part of the puzzle.
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-01T04:07:53.963Z
Comments.Text: What does a subtitle mean for a silent film?  Silent films don't have spoken dialogue.  If the actors are not speaking what is there to put in a subtitle?  Are you asking how to summarize the actions of video segments into words?  If that is the case you may find the following video of interest although it does not directly address your question: https://www.youtube.com/watch?time_continue=1&v=AR3hY9iB5-I
Comments.UserDisplayName: Brian O'Donnell
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-01T04:26:12.493Z
Comments.Text: Check out Amazon's Rekognition which combines algorithms like object detection, tracking,facial recognition, activity detection, and celebrity recognition: https://aws.amazon.com/rekognition/ and https://www.youtube.com/watch?v=SNONL4IecHE
Comments.UserDisplayName: Brian O'Donnell
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-01T22:17:50.353Z
Comments.Text: Added a clarification
Comments.UserDisplayName: yoav.aviram
Tags: <image-recognition>
Title: Create captions based on a series of images
UserDisplayName: yoav.aviram
ThreadText: # Create captions based on a series of images
  
  (Asked by: username_0 on 2018-02-28)
  
  I'd like to generate subtitles for a silent film. Is there an open source project out there capable of creating captions based on a series of images (such as a scene from a movie)?
  
  EDIT: thanks for the comments below. To clarify, what i'm looking for is an algorithm which can generate a caption for a sequences of images within a movie describing what happens in the sequence. This is for preliminary research, so accuracy is less important.
  
  **Tags:** image-recognition
  
  > **Comments:**
  > **username_1**: This is a really cool question.  My guess, based on current limitations, is that there is no AI in existence that could come close to providing this function... today.  But even as AI advances, this particular function will be challenging.  The AI has to understand the context AND subtext of the silent scenes.  *(Many humans seem not to get subtext;)*  Both are far beyond current capabilities, but I'm sure facial recognition related to emotions will be a major part of the puzzle.  
  > **username_2**: What does a subtitle mean for a silent film?  Silent films don't have spoken dialogue.  If the actors are not speaking what is there to put in a subtitle?  Are you asking how to summarize the actions of video segments into words?  If that is the case you may find the following video of interest although it does not directly address your question: https://www.youtube.com/watch?time_continue=1&v=AR3hY9iB5-I  
  > **username_2**: Check out Amazon's Rekognition which combines algorithms like object detection, tracking,facial recognition, activity detection, and celebrity recognition: https://aws.amazon.com/rekognition/ and https://www.youtube.com/watch?v=SNONL4IecHE  
  > **username_0**: Added a clarification  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2018-09-01. Score: 1 | Accepted answer
  
  What you might want to look for is called **video captioning**. Earlier examples from this line of research is:
  
  * Venugopalan et. al. [Sequence to Sequence – Video to Text](https://arxiv.org/pdf/1505.00487.pdf), in ICCV 2015
  * Pasunuru et.al. [Multi-Task Video Captioning with Video and
    Entailment Generation](http://aclweb.org/anthology/P/P17/P17-1117.pdf), in ACL 2017
  
  Below is a screenshot of the results (positive and negative) reported in those papers:
  [![Example from Venugopalan1](https://i.sstatic.net/neTfw.png)](https://i.sstatic.net/neTfw.png)
  [![Example from Pasunuru](https://i.sstatic.net/qO1XT.png)](https://i.sstatic.net/qO1XT.png)
  
  For the ICCV paper, it's not hard to find some implementations, e.g. [here](https://github.com/vijayvee/video-captioning).
  
  For more recent results, I would suggest to look into the [ActivityNet 2017 Challenge - dense captioning](http://activity-net.org/challenges/2017/captioning.html) or its [2018 version](http://activity-net.org/challenges/2018/tasks/anet_captioning.html). Some winning solutions include:
  
  * [Jointly Localizing and Describing Events for Dense Video Captioning](https://arxiv.org/pdf/1804.08274.pdf)
  * [RUC+CMU: System Report for Dense Captioning Events in Videos](https://arxiv.org/pdf/1806.08854.pdf)
  
  However I am not sure whether any open-source implementation has been released.
  
  
  ---
  
  ### Answer by username_4 on 2018-03-03. Score: 0
  
  Here is one open-source implementation. [Temporal Tessellation: A Unified Approach for Video Analysis](https://github.com/dot27/temporal-tessellation)
  
  For more you can dig into some research publications and see if they give a link to their implementation. Most researchers put up their work for public.
  
  Here is a list of publications that present their work related to video captioning using machine learning. [Awesome Deep Vision](https://github.com/kjw0612/awesome-deep-vision#video-captioning)
  
  Here is another publication that shows how to generate captions for videos. [Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks](https://www.ics.uci.edu/~yyang8/research/video-caption/video-caption.pdf)
  
  
  ---
  
  ### Answer by username_5 on 2018-07-01. Score: 0
  
  No , even generating brief summary of a video is out of scope as of current state of the art. Training such a model is a tough task. Video understanding is pretty far from the research as of now.
  But you could try generating descriptions of some keyframes of the video , and align them to form meaningful passages.
  
  check this <https://arxiv.org/abs/1611.06607>
  
  
  ---
  
  