Answers.Body: <p>My two cents on this topic:</p>
  <p><em>&quot;After all, &quot;color&quot; is just another statistic&quot;</em>, I think this is the simple (and correct) answer to the question. To go a bit deeper, you can check this <a href="https://arxiv.org/pdf/1701.01036.pdf" rel="nofollow noreferrer">paper</a>, which shows how <strong>minimizing a loss based on the Gram matrix is mathematically equivalent to minimizing the <a href="https://en.wikipedia.org/wiki/Kernel_embedding_of_distributions#Measuring_distance_between_distributions" rel="nofollow noreferrer">Maximum Mean Discrepancy</a> between the inputs and targets distribution</strong>. The two distributions inevitably contains information about colors, so while disentangle spatial features is rather simple (you could simply show one pixel at the time instead of an image),  disentangling colors is much more tricky, cause it's an intrinsic characteristic of each point.</p>
  <p>A last remark from my side is that the main problem when working with style transfer is precisely that &quot;style&quot; mean everything. This is not a problem for papers that simply try to achieve it per se, i.e. without a real use case in mind, but it becomes fundamental in real applications. A concrete example of this is super resolution. Many papers try to achieve it with style transfer, coupling low resolution and high resolution images. Ideally the features you would like to transfer are enhanced sharpness and maybe texture injection for details generation. Problem is that along with them there are always side features that hinder the quality of the resulting image, among which noise specific to the target domain, and also colors.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-24T13:03:33.320Z
Answers.LastActivityDate: 2021-08-24T13:03:33.320Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>If you want to put color channel value for the loss function in neural network training, I have seen Structural Similarity Index Measure, SSIM, being used at Super Resolution neural network. Here is one webpage about <a href="https://www.imatest.com/support/docs/23-1/ssim/#:%7E:text=Introduction%20%E2%80%94%20The%20Structural%20Similarity%20Index,by%20losses%20in%20data%20transmission." rel="nofollow noreferrer">SSIM</a> and example code from <a href="https://scikit-image.org/docs/stable/auto_examples/transform/plot_ssim.html" rel="nofollow noreferrer">Sci-Kit Image</a>.</p>
  <p>Your intention looks like not just one pixel but surrounding, so you may somehow putting moving window or Region of Interest in SSIM equation. Or you have to write own equation using Region of Interest with pixel value(s).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-13T19:10:41.463Z
Answers.LastActivityDate: 2023-09-13T19:10:41.463Z
Answers.UserDisplayName: Cloud Cho
Body: <p>I've recently been reading a lot about style transfer, its applications and implications. I understand what the Gram matrix is and does. I can program it. But one thing that has been boggling me is: how does the VGG style loss incorporate color information into the style?</p>
  <p>In the paper &quot;<a href="https://proceedings.neurips.cc/paper/2015/file/a5e00132373a7031000fd987a3c9f87b-Paper.pdf" rel="nofollow noreferrer">Texture Synthesis by CNNs</a>&quot;, Gatys et al. show that minimizing the MSE between the Gram matrices of a random white noise image and a &quot;target texture&quot; yields new instances of that texture, with stochastic variation. I understand that this must work, as the Gram matrix measures the correlation between features detected by the VGG activations across channels, without spatial relation. So if we optimize the white noise image to have the same Gram matrix, it will exhibit the same statistics, and hence look like an instance of the original texture.</p>
  <p>But how does this work with color? Of course, the VGG could learn something like a mean filter, with all ones, whose output would be the avg. color over that filter kernel. After all, &quot;color&quot; is just another statistic. But then when using that in conjunction with the Gram loss, wouldn't this information be lost, as it's all just correlation and hence &quot;relative&quot; to each other?</p>
  <p>While writing this question, I'm starting to think of it like this: Maybe the feature correlation expresses these color constraints in some form like: &quot;if one part is red, there must be a green part close to it&quot; (for the radish), or &quot;if there is a rounded edge, one side of it must be in shadow (=darker)&quot; in case of the stone texture. This would tie color to the surrounding statistics (e.g., edges, other colors) and is the only reason I can think of why this works at all.</p>
  <p>Can somebody confirm/refute this, and share their thoughts? Happy to discuss!</p>
  <p><a href="https://i.sstatic.net/Bw87C.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Bw87C.jpg" alt="Image Source: Gatys et al., Texture Synthesis by Convolutional Neural Networks" /></a></p>
  <p>Image Source: Texture Synthesis by Convolutional Neural Networks, Gatys et al.</p>
  
Tags: <convolutional-neural-networks><image-generation><statistics><vgg><style-transfer>
Title: How does a VGG-based Style-Loss incorporate color information?
UserDisplayName: masterBroesel
ThreadText: # How does a VGG-based Style-Loss incorporate color information?
  
  (Asked by: username_0 on 2021-08-24)
  
  I've recently been reading a lot about style transfer, its applications and implications. I understand what the Gram matrix is and does. I can program it. But one thing that has been boggling me is: how does the VGG style loss incorporate color information into the style?
  
  In the paper "[Texture Synthesis by CNNs](https://proceedings.neurips.cc/paper/2015/file/a5e00132373a7031000fd987a3c9f87b-Paper.pdf)", Gatys et al. show that minimizing the MSE between the Gram matrices of a random white noise image and a "target texture" yields new instances of that texture, with stochastic variation. I understand that this must work, as the Gram matrix measures the correlation between features detected by the VGG activations across channels, without spatial relation. So if we optimize the white noise image to have the same Gram matrix, it will exhibit the same statistics, and hence look like an instance of the original texture.
  
  But how does this work with color? Of course, the VGG could learn something like a mean filter, with all ones, whose output would be the avg. color over that filter kernel. After all, "color" is just another statistic. But then when using that in conjunction with the Gram loss, wouldn't this information be lost, as it's all just correlation and hence "relative" to each other?
  
  While writing this question, I'm starting to think of it like this: Maybe the feature correlation expresses these color constraints in some form like: "if one part is red, there must be a green part close to it" (for the radish), or "if there is a rounded edge, one side of it must be in shadow (=darker)" in case of the stone texture. This would tie color to the surrounding statistics (e.g., edges, other colors) and is the only reason I can think of why this works at all.
  
  Can somebody confirm/refute this, and share their thoughts? Happy to discuss!
  
  [![Image Source: Gatys et al., Texture Synthesis by Convolutional Neural Networks](https://i.sstatic.net/Bw87C.jpg)](https://i.sstatic.net/Bw87C.jpg)
  
  Image Source: Texture Synthesis by Convolutional Neural Networks, Gatys et al.
  
  **Tags:** convolutional-neural-networks, image-generation, statistics, vgg, style-transfer
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-08-24. Score: 0
  
  My two cents on this topic:
  
  *"After all, "color" is just another statistic"*, I think this is the simple (and correct) answer to the question. To go a bit deeper, you can check this [paper](https://arxiv.org/pdf/1701.01036.pdf), which shows how **minimizing a loss based on the Gram matrix is mathematically equivalent to minimizing the [Maximum Mean Discrepancy](https://en.wikipedia.org/wiki/Kernel_embedding_of_distributions#Measuring_distance_between_distributions) between the inputs and targets distribution**. The two distributions inevitably contains information about colors, so while disentangle spatial features is rather simple (you could simply show one pixel at the time instead of an image), disentangling colors is much more tricky, cause it's an intrinsic characteristic of each point.
  
  A last remark from my side is that the main problem when working with style transfer is precisely that "style" mean everything. This is not a problem for papers that simply try to achieve it per se, i.e. without a real use case in mind, but it becomes fundamental in real applications. A concrete example of this is super resolution. Many papers try to achieve it with style transfer, coupling low resolution and high resolution images. Ideally the features you would like to transfer are enhanced sharpness and maybe texture injection for details generation. Problem is that along with them there are always side features that hinder the quality of the resulting image, among which noise specific to the target domain, and also colors.
  
  
  ---
  
  ### Answer by username_2 on 2023-09-13. Score: 0
  
  If you want to put color channel value for the loss function in neural network training, I have seen Structural Similarity Index Measure, SSIM, being used at Super Resolution neural network. Here is one webpage about [SSIM](https://www.imatest.com/support/docs/23-1/ssim/#:%7E:text=Introduction%20%E2%80%94%20The%20Structural%20Similarity%20Index,by%20losses%20in%20data%20transmission.) and example code from [Sci-Kit Image](https://scikit-image.org/docs/stable/auto_examples/transform/plot_ssim.html).
  
  Your intention looks like not just one pixel but surrounding, so you may somehow putting moving window or Region of Interest in SSIM equation. Or you have to write own equation using Region of Interest with pixel value(s).
  
  
  ---
  
  