Body: <p>I am a software engineer who is quickly ramping up on AI tech, but am nevertheless very new to the sector.</p>
  <p>A collegue has an extensive collection of training videos, the vertical is wheelchair seating and mobility and the training content are the mechanical hardware skills gained over a 30 year career.  I am telling him that he needs to be the first seating vendor on the block with an AI that has incorporated into its context everything he has included in his video series and is able to generate training videos of its own.</p>
  <p>Stated another way, I'm looking for a way for an AI to learn the skills my friend teaches not only through verbal prompts, but through hands-on demonstrations.</p>
  <p>Although he is very precise in his presentations and one approach could be to pull text from the audio and normalize it for input into training an OpenAI model, I can't help but think that an AI should have full access to the videos, including the visual portions and should be able to generate the same kind of output.</p>
  <p>If we can have AIs that can generate entertainment videos with a textual input, couldn't we have AIs that generate full training videos with textual and video inputs?</p>
  <p>Perhaps some kind of time code correlated input with screen grabs from the training videos presented in training alongside with text scraped/recognized from the audio track.</p>
  <p>I hear that GPT4 is able to train from correlated text/images but I'm having a hard time finding more information about this.</p>
  <p>The owner wants to be able to validate/challenge the AI and correct any errors in the resulting output (audio and/or video).  My OpenAI API reading indicates that at least for text based AIs this is now possible.</p>
  <p>Is current technology up to this task?   How would a content creator correct an AI that produces incorrect video output ?</p>
  <p>Exemplar:
  <a href="https://youtu.be/hhgEBm7C2G8" rel="nofollow noreferrer">https://youtu.be/hhgEBm7C2G8</a></p>
  <p>Thanks !</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-29T11:56:44.417Z
Comments.Text: Learn the graphic design and programme language to using AI to create training virtual anime video
Comments.UserDisplayName: Farha Kousar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-29T20:04:46.983Z
Comments.Text: You are a a year or three too early. Best options for video generation today are not up to scratch for such a task. Some creatives working in video could make use of today's image, text, voice and video generation for effect, and sometimes for efficiency. But is still a way to go before it could be core to any decent prodtion
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-29T20:08:21.233Z
Comments.Text: Examples of current cutting edge videos using AI gen: https://m.youtube.com/watch?v=saZtmlJYagE
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-06T22:06:30.460Z
Comments.Text: Checking back in, I'm aware of Synthesia and Heygen doing script-driven AI animations (with collateral like background video and pictures added to the scene)>
  
  I'm also aware of AIs that generate small video clips from textual or photo prompts: https://stability.ai/stable-video
  
  What would be the next step to have an AI understand the correlations between an audio track and the vocational skill being shown?  
  
  Can anybody see a roadmap here?
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-20T23:48:38.030Z
Comments.Text: I see that GPT4o has been released, which apparently can consume and understand video.  I'm looking at whether this model gets closer to what I need to do as described in this article.
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-30T18:39:35.160Z
Comments.Text: I also see NVIDIA and OpenAI Sora being models that have some understanding of world physics.  Could these be used in conjunction with GPT4o to achieve the stated goal?
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-03T13:23:32.080Z
Comments.Text: Looking even farther forward, I can envision autocreation of training video content and 3D artifacts to populate a 3D assessment space where students must master and demonstrate learned concepts where the results are judged in a virtual space against a gold standard, but otherwise allowing students to emulate a real shop in the virtual space for the particular skills(s) being taught.
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-06T09:53:59.410Z
Comments.Text: This research shows some promise in   better enabling multimodal time series analysis : https://techxplore.com/news/2024-07-brain-artificial-dendritic-neural-circuit.html?utm_source=nwletter&utm_medium=email&utm_campaign=daily-nwletter
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-06T09:57:32.733Z
Comments.Text: Also interesting - adding ambient audio to training processes - gong beyond simple text to speech and paying attention to the sounds made by performing some vocational tasks allows them to be performed more correctly and quickly : https://techxplore.com/news/2024-07-adding-audio-robots-job.html?utm_source=nwletter&utm_medium=email&utm_campaign=daily-nwletter
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-04T11:00:52.617Z
Comments.Text: I’ll check out GPT4o’s ability to accept both video and audio input and see how well it correlates these modalities:  https://cookbook.openai.com/examples/gpt4o/introduction_to_gpt4o
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-10T00:43:45.463Z
Comments.Text: I've made some progress, but it's slow going as gpt4o appears to have trouble reasoning beyond the surface.  What techniques could I use to better help the AI correlate multimodal text and image inputs across time (as not every input is localized, sometimes a newer subtitle helps to unlock the meaning of an older image in a video, and vice versa) ?
  
  More details:
  
  https://community.openai.com/t/having-trouble-in-advanced-multimodal-reasoning-beyond-the-surface/1011106
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-10T00:48:17.060Z
Comments.Text: I'm also aware that fine tuning is now supported on gpt4o, but only textual training data sets are allowed, so I want to train on multimodal inputs I'd have to make sure that images are properly reduced to text descriptions.  This is trickier than it sounds as there are a mountain of corner cases.  I'm trying my hand at manually adding training details to input context to produce better and more nuanced and sufficiently detailed image summaries that are useful for the purpose (distilling training details from the vocational training video) but the human effort mounts and AI performance lacks.
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-11T19:25:57.457Z
Comments.Text: I switched the entire factory from OpenAi GPT4o and am having markedly better results. It was a pleasure to move an “AI on Botox” to an AI that showed a deeper level of nuanced understanding of the problem space. Now to figure out how to fine tune Claude Sonnet 3.5. Apparently I’ll need to invest in some beefy hardware as I don’t think Anthropic has a fine tuning API exposed like OpenAI has.
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-21T15:22:45.143Z
Comments.Text: I've had some good success with Sonnet, please see https://dev.to/mdear/inceptive-techniques-to-pull-summaries-and-e-learning-out-of-vocational-training-video-series-25eo
  
  and
  
  https://gist.github.com/mdear/4cd8ee0bfb807840c64b34a6569a8949
  
  @NeilSlater
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-30T13:11:23.853Z
Comments.Text: I'm now considering using laion/CLIP-ViT-H-14-laion2B-s32B-b79K OpenCLIP for embeddings against Pinecone for natural language semantic search access to the growing corpus of AI summaries and e-learning content generated from this video series as part of a RAG architecture.
Comments.UserDisplayName: lukabloomrox
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-31T18:50:31.893Z
Comments.Text: I think I may move towards BLIP and pre-translate any non-English search text into English as BLIP and BLIP2 purportedly better aligns objects represented by both text and images into a single aligned embedding, whereas CLIP can only accept either text or images and puts them in a semantically similar space.  I'm looking for a model that can understand the relationships among the modalities to produce a better embedding that incorporates aspects of all modalities including cross-understanding (using text to inform image understanding and vice versa).
Comments.UserDisplayName: lukabloomrox
Tags: <training><multilayer-perceptrons><multi-task-learning>
Title: How to generate original training videos based on existing videoset?
UserDisplayName: lukabloomrox
ThreadText: # How to generate original training videos based on existing videoset?
  
  (Asked by: username_0 on 2023-06-08)
  
  I am a software engineer who is quickly ramping up on AI tech, but am nevertheless very new to the sector.
  
  A collegue has an extensive collection of training videos, the vertical is wheelchair seating and mobility and the training content are the mechanical hardware skills gained over a 30 year career. I am telling him that he needs to be the first seating vendor on the block with an AI that has incorporated into its context everything he has included in his video series and is able to generate training videos of its own.
  
  Stated another way, I'm looking for a way for an AI to learn the skills my friend teaches not only through verbal prompts, but through hands-on demonstrations.
  
  Although he is very precise in his presentations and one approach could be to pull text from the audio and normalize it for input into training an OpenAI model, I can't help but think that an AI should have full access to the videos, including the visual portions and should be able to generate the same kind of output.
  
  If we can have AIs that can generate entertainment videos with a textual input, couldn't we have AIs that generate full training videos with textual and video inputs?
  
  Perhaps some kind of time code correlated input with screen grabs from the training videos presented in training alongside with text scraped/recognized from the audio track.
  
  I hear that GPT4 is able to train from correlated text/images but I'm having a hard time finding more information about this.
  
  The owner wants to be able to validate/challenge the AI and correct any errors in the resulting output (audio and/or video). My OpenAI API reading indicates that at least for text based AIs this is now possible.
  
  Is current technology up to this task? How would a content creator correct an AI that produces incorrect video output ?
  
  Exemplar:
  <https://youtu.be/hhgEBm7C2G8>
  
  Thanks !
  
  **Tags:** training, multilayer-perceptrons, multi-task-learning
  > **Comments:**
  > **username_1**: Learn the graphic design and programme language to using AI to create training virtual anime video  
  > **username_2**: You are a a year or three too early. Best options for video generation today are not up to scratch for such a task. Some creatives working in video could make use of today's image, text, voice and video generation for effect, and sometimes for efficiency. But is still a way to go before it could be core to any decent prodtion  
  > **username_2**: Examples of current cutting edge videos using AI gen: https://m.youtube.com/watch?v=saZtmlJYagE  
  > **username_0**: Checking back in, I'm aware of Synthesia and Heygen doing script-driven AI animations (with collateral like background video and pictures added to the scene)>  I'm also aware of AIs that generate small video clips from textual or photo prompts: https://stability.ai/stable-video  What would be the next step to have an AI understand the correlations between an audio track and the vocational skill being shown?    Can anybody see a roadmap here?  
  > **username_0**: I see that GPT4o has been released, which apparently can consume and understand video.  I'm looking at whether this model gets closer to what I need to do as described in this article.  
  > **username_0**: I also see NVIDIA and OpenAI Sora being models that have some understanding of world physics.  Could these be used in conjunction with GPT4o to achieve the stated goal?  
  > **username_0**: Looking even farther forward, I can envision autocreation of training video content and 3D artifacts to populate a 3D assessment space where students must master and demonstrate learned concepts where the results are judged in a virtual space against a gold standard, but otherwise allowing students to emulate a real shop in the virtual space for the particular skills(s) being taught.  
  > **username_0**: This research shows some promise in   better enabling multimodal time series analysis : https://techxplore.com/news/2024-07-brain-artificial-dendritic-neural-circuit.html?utm_source=nwletter&utm_medium=email&utm_campaign=daily-nwletter  
  > **username_0**: Also interesting - adding ambient audio to training processes - gong beyond simple text to speech and paying attention to the sounds made by performing some vocational tasks allows them to be performed more correctly and quickly : https://techxplore.com/news/2024-07-adding-audio-robots-job.html?utm_source=nwletter&utm_medium=email&utm_campaign=daily-nwletter  
  > **username_0**: I’ll check out GPT4o’s ability to accept both video and audio input and see how well it correlates these modalities:  https://cookbook.openai.com/examples/gpt4o/introduction_to_gpt4o  
  > **username_0**: I've made some progress, but it's slow going as gpt4o appears to have trouble reasoning beyond the surface.  What techniques could I use to better help the AI correlate multimodal text and image inputs across time (as not every input is localized, sometimes a newer subtitle helps to unlock the meaning of an older image in a video, and vice versa) ?  More details:  https://community.openai.com/t/having-trouble-in-advanced-multimodal-reasoning-beyond-the-surface/1011106  
  > **username_0**: I'm also aware that fine tuning is now supported on gpt4o, but only textual training data sets are allowed, so I want to train on multimodal inputs I'd have to make sure that images are properly reduced to text descriptions.  This is trickier than it sounds as there are a mountain of corner cases.  I'm trying my hand at manually adding training details to input context to produce better and more nuanced and sufficiently detailed image summaries that are useful for the purpose (distilling training details from the vocational training video) but the human effort mounts and AI performance lacks.  
  > **username_0**: I switched the entire factory from OpenAi GPT4o and am having markedly better results. It was a pleasure to move an “AI on Botox” to an AI that showed a deeper level of nuanced understanding of the problem space. Now to figure out how to fine tune Claude Sonnet 3.5. Apparently I’ll need to invest in some beefy hardware as I don’t think Anthropic has a fine tuning API exposed like OpenAI has.  
  > **username_0**: I've had some good success with Sonnet, please see https://dev.to/mdear/inceptive-techniques-to-pull-summaries-and-e-learning-out-of-vocational-training-video-series-25eo  and  https://gist.github.com/mdear/4cd8ee0bfb807840c64b34a6569a8949  @NeilSlater  
  > **username_0**: I'm now considering using laion/CLIP-ViT-H-14-laion2B-s32B-b79K OpenCLIP for embeddings against Pinecone for natural language semantic search access to the growing corpus of AI summaries and e-learning content generated from this video series as part of a RAG architecture.  
  > **username_0**: I think I may move towards BLIP and pre-translate any non-English search text into English as BLIP and BLIP2 purportedly better aligns objects represented by both text and images into a single aligned embedding, whereas CLIP can only accept either text or images and puts them in a semantically similar space.  I'm looking for a model that can understand the relationships among the modalities to produce a better embedding that incorporates aspects of all modalities including cross-understanding (using text to inform image understanding and vice versa).  
  
  ---
  
  ## 0 answers
  
  