Answers.Body: <p>The approach you outline in the steps 1 to 4 will work to a degree, and may be used in some very basic video upscaling systems. However, the results may not be satisfcatory. A very typical problem will be flickering due to inconsistent upscaling choices made on consecutive frames. In addition, upscaling and interpolating the spatial dimensions of a video sequence will increase the pixel &quot;jumps&quot; between frames (more pixels will have large change between each frame), so to keep the output visually smooth it is often desirable to increase time resolution, i.e. increase the frame rate, interpolating smartly between frames.</p>
  <p>The first problem, differences in details when processing frames separately, is called temporal coherence. There are a variety of ways to increase temporal coherence when upscaling. All require some comparison between consecutive frame images. Some are specified as soft constraints, or a loss metric applied to the processing during upscaling. Many models are trained specifically for video upscaling, taking multiple input frames, so that such losses can be used to improve the core model.</p>
  <p>Sometimes, temporal coherence can be implemented as a modifier when recombining the upscaled frames - i.e. as part of your suggested step 3. This will be ok for simple interpolation upscaling that doesn't attempt to add missing details. It will tend towards smooth blurring for the upscales, similar to the simplest upscale options in image editors.</p>
  <p>This is still an ongoing area of research, with papers routinely published. Here are two that I found on a quick search. I have not vetted these papers for whether they might be useful to you, they are just examples:</p>
  <ul>
  <li><a href="https://arxiv.org/abs/2312.06640" rel="nofollow noreferrer">Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution (Decembver 2023)</a></li>
  <li><a href="https://www.sciencedirect.com/science/article/abs/pii/S0952197623019735" rel="nofollow noreferrer">Fine-grained video super-resolution via spatial-temporal learning and image detail enhancement (May 2024 preview)</a></li>
  </ul>
  <p>The second problem, frame rate increase, is solved by having models specifically built to solve that problem. Again this is usually achieved by having inputs of consecutive frames that need interpolating, but this time output will be an interstitial frame that is not an upscale of either input. Training data for these models is typically high frame rate videos where the input has had frames removed, and the output is the relevant missing frames.</p>
  <p>Some video generation models perform both tasks - image and frame rate upscaling - starting with an initial small low frame rate generation, and alternating layers between frame rate increase and image size increase.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T14:56:42.293Z
Answers.Text: Thank you! The “physics of the process” and related problems are clear to me. But the block diagrams and mathematics underlying the methods for increasing temporal coherence and increasing frame rates need to be studied separately. Because, as I understand it, GAN in its pure form is not used for this. Therefore, surely those models that solve both problems have a specific structure?
Answers.UserDisplayName: ayr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T14:58:06.823Z
Answers.Text: Well, I think that using the keywords from your answer I will find enough information to study.
Answers.UserDisplayName: ayr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T15:29:25.737Z
Answers.Text: @dtn: There are quite a few different approaches, and different kinds of models used to solve these problems. So there isn't a specific structure, it's more about how deep you want to look into the issue. Similar is true for image upscaling in the first place. A simple model is to linearly interpolate to get values for missing pixels. Slightly more sophisticated is to use bicubic interpolation. Wavelet decomposition. The modern AI-based upscalers don't do any of that, and even looking at the generative model upscalers, there are GAN-based, Diffusion-based etc. No single model or structure
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-18T05:26:11.183Z
Answers.Text: In general, I would like to study the issue at such a level that I could experiment with the upscale of some old films, which are now perceived completely differently than before... Well, and get some results and practical knowledge.
Answers.UserDisplayName: ayr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-18T08:53:02.640Z
Answers.Text: @dtn I suggest you start with a naive system based around your initial idea. Building it and experiencing the issue with temporal coherence and frame rate first hand will be valuable experience, and far more rewarding than trying to build the most sophisticated version you read about as the first attempt. Once you have a basic version, then you can refer to ideas in this answer to upgrade it
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-19T03:46:49.723Z
Answers.Text: I think you're right. I can start with short films. To gain basic skills. And the rest of the base will be added in the process. Thank you for your response and for your time!
Answers.UserDisplayName: ayr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T11:15:28.803Z
Answers.LastActivityDate: 2024-01-17T12:24:50.180Z
Answers.LastEditDate: 2024-01-17T12:24:50.180Z
Answers.UserDisplayName: Neil Slater
Body: <p>With the help of artificial intelligence, it is possible to increase the resolution of images that are initially low resolution, bringing it to ultra-high resolution. Also, initially static images are turned into short animated videos. But I'm interested in AI upscaling of video materials such as old movies and perhaps cartoons.</p>
  <p>In my opinion, the technology is like this:</p>
  <ol>
  <li>video file is dismantled, i.e. is decomposed into separate pictures-frames.</li>
  <li>using GAN, the resolution of each frame is brought to the required value.</li>
  <li>new frames are re-edited, i.e. new pictures-frames are combined into a new video file.</li>
  <li>as initial data for training the neural network, the original film or cartoon, images with similar characteristics, other films or cartoons, and a large group of high-resolution images similar to frames are used.</li>
  </ol>
  <p>Do I understand the technology of AI upscaling of films and cartoons correctly, or are there some other nuances and important details that I forgot to mention? I would like to get a clearer.</p>
  
Tags: <deep-learning><generative-adversarial-networks><image-processing><image-super-resolution>
Title: Do I understand the technology of AI upscaling of films and cartoons correctly?
UserDisplayName: ayr
ThreadText: # Do I understand the technology of AI upscaling of films and cartoons correctly?
  
  (Asked by: username_0 on 2024-01-17)
  
  With the help of artificial intelligence, it is possible to increase the resolution of images that are initially low resolution, bringing it to ultra-high resolution. Also, initially static images are turned into short animated videos. But I'm interested in AI upscaling of video materials such as old movies and perhaps cartoons.
  
  In my opinion, the technology is like this:
  
  1. video file is dismantled, i.e. is decomposed into separate pictures-frames.
  2. using GAN, the resolution of each frame is brought to the required value.
  3. new frames are re-edited, i.e. new pictures-frames are combined into a new video file.
  4. as initial data for training the neural network, the original film or cartoon, images with similar characteristics, other films or cartoons, and a large group of high-resolution images similar to frames are used.
  
  Do I understand the technology of AI upscaling of films and cartoons correctly, or are there some other nuances and important details that I forgot to mention? I would like to get a clearer.
  
  **Tags:** deep-learning, generative-adversarial-networks, image-processing, image-super-resolution
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-17. Score: 3 | Accepted answer
  
  The approach you outline in the steps 1 to 4 will work to a degree, and may be used in some very basic video upscaling systems. However, the results may not be satisfcatory. A very typical problem will be flickering due to inconsistent upscaling choices made on consecutive frames. In addition, upscaling and interpolating the spatial dimensions of a video sequence will increase the pixel "jumps" between frames (more pixels will have large change between each frame), so to keep the output visually smooth it is often desirable to increase time resolution, i.e. increase the frame rate, interpolating smartly between frames.
  
  The first problem, differences in details when processing frames separately, is called temporal coherence. There are a variety of ways to increase temporal coherence when upscaling. All require some comparison between consecutive frame images. Some are specified as soft constraints, or a loss metric applied to the processing during upscaling. Many models are trained specifically for video upscaling, taking multiple input frames, so that such losses can be used to improve the core model.
  
  Sometimes, temporal coherence can be implemented as a modifier when recombining the upscaled frames - i.e. as part of your suggested step 3. This will be ok for simple interpolation upscaling that doesn't attempt to add missing details. It will tend towards smooth blurring for the upscales, similar to the simplest upscale options in image editors.
  
  This is still an ongoing area of research, with papers routinely published. Here are two that I found on a quick search. I have not vetted these papers for whether they might be useful to you, they are just examples:
  
  * [Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution (Decembver 2023)](https://arxiv.org/abs/2312.06640)
  * [Fine-grained video super-resolution via spatial-temporal learning and image detail enhancement (May 2024 preview)](https://www.sciencedirect.com/science/article/abs/pii/S0952197623019735)
  
  The second problem, frame rate increase, is solved by having models specifically built to solve that problem. Again this is usually achieved by having inputs of consecutive frames that need interpolating, but this time output will be an interstitial frame that is not an upscale of either input. Training data for these models is typically high frame rate videos where the input has had frames removed, and the output is the relevant missing frames.
  
  Some video generation models perform both tasks - image and frame rate upscaling - starting with an initial small low frame rate generation, and alternating layers between frame rate increase and image size increase.
  
  > **Comments:**
  > **username_0**: Thank you! The “physics of the process” and related problems are clear to me. But the block diagrams and mathematics underlying the methods for increasing temporal coherence and increasing frame rates need to be studied separately. Because, as I understand it, GAN in its pure form is not used for this. Therefore, surely those models that solve both problems have a specific structure?  
  > **username_0**: Well, I think that using the keywords from your answer I will find enough information to study.  
  > **username_1**: @dtn: There are quite a few different approaches, and different kinds of models used to solve these problems. So there isn't a specific structure, it's more about how deep you want to look into the issue. Similar is true for image upscaling in the first place. A simple model is to linearly interpolate to get values for missing pixels. Slightly more sophisticated is to use bicubic interpolation. Wavelet decomposition. The modern AI-based upscalers don't do any of that, and even looking at the generative model upscalers, there are GAN-based, Diffusion-based etc. No single model or structure  
  > **username_0**: In general, I would like to study the issue at such a level that I could experiment with the upscale of some old films, which are now perceived completely differently than before... Well, and get some results and practical knowledge.  
  > **username_1**: @dtn I suggest you start with a naive system based around your initial idea. Building it and experiencing the issue with temporal coherence and frame rate first hand will be valuable experience, and far more rewarding than trying to build the most sophisticated version you read about as the first attempt. Once you have a basic version, then you can refer to ideas in this answer to upgrade it  
  > **username_0**: I think you're right. I can start with short films. To gain basic skills. And the rest of the base will be added in the process. Thank you for your response and for your time!  
  
  ---
  
  