Answers.Body: <p>When the <a href="https://arxiv.org/abs/1508.06576" rel="nofollow noreferrer">original Neural Transfer paper</a> was published, your question stayed unanswered for a while.  The reason why the <a href="https://en.wikipedia.org/wiki/Gram_matrix" rel="nofollow noreferrer">Gram matrix</a> represents artistic style was not entirely clear. A satisfactory (in my opinion) explanation came with the <a href="https://arxiv.org/abs/1701.01036" rel="nofollow noreferrer"><em>&quot;Demystifying Neural Style Transfer&quot;</em></a> paper.</p>
  <p>The basic idea is that you cannot just directly compare activations for two images. The spatial positions of various features are different for the source and target image, so you should somehow get rid of positional information of the activations and compare their <em>distributions</em> across the whole image. The goal of the style transfer task is thus to make the two distributions as close as possible. One of the possible measures of distance between two distributions <span class="math-container">$P$</span> and <span class="math-container">$Q$</span> is the <a href="https://en.wikipedia.org/wiki/Kernel_embedding_of_distributions#Kernel_two-sample_test" rel="nofollow noreferrer">Maximum Mean Distance</a> (MMD):</p>
  <p><span class="math-container">$$\text{MMD}^2(P,Q) = \left\Vert \mathbb{E}_{P}[\phi(X)] - \mathbb{E}_{Q}[\phi(Y)]\right\Vert^2$$</span></p>
  <p>With <span class="math-container">$\phi(\cdot)$</span> being a feature function - in our case it would be the NN activations in a particular layer. The next step would be to apply a &quot;<a href="https://en.wikipedia.org/wiki/Kernel_method" rel="nofollow noreferrer">kernel trick</a>&quot; to the MMD, representing it through a kernel <span class="math-container">$k(x,y)$</span></p>
  <p><span class="math-container">$$\text{MMD}^2(P,Q) = \mathbb{E}[k(x_i,x_j)] + \mathbb{E}[k(y_i,y_j)]  - 2 \mathbb{E}[k(x_i,y_j)]$$</span></p>
  <p>The Gram matrix of the original style transfer corresponds to the squared dot-product kernel <span class="math-container">$k(x,y) = (x^Ty)^2$</span>:
  <span class="math-container">$$\text{MMD}^2(P,Q) =  \mathbb{E}[(x^T_ix_j)^2] + \mathbb{E}[(y^T_iy_j)^2]  - 2 \mathbb{E}[(x^T_iy_j)^2]  =  \left\Vert G^x - G^y\right\Vert_F^2 $$</span></p>
  <p>Where <span class="math-container">$G^x$</span> and <span class="math-container">$G^y$</span> are the Gram matrices for activations and <span class="math-container">$\Vert\cdot\Vert_F$</span> is the <a href="https://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm" rel="nofollow noreferrer">Frobenius norm</a>. The last equality gives a bit of trouble to those not used to the kernel trick, so I'll expand it (assuming <a href="https://en.wikipedia.org/wiki/Einstein_notation" rel="nofollow noreferrer">Einstein summation</a> convention):</p>
  <p><span class="math-container">$$\begin{array}{l}
   \mathbb{E}[(x^T_ix_j)^2] + \mathbb{E}[(y^T_iy_j)^2]  - 2 \mathbb{E}[(x^T_iy_j)^2]  =  \\
   = x_{ik}x_{jk}x_{im}x_{jm} + y_{ik}y_{jk}y_{im}y_{jm} - 2  x_{ik}x_{jk}y_{im}y_{jm} \\
   = (x_{ik}x_{im} - y_{ik}y_{im})(x_{jk}x_{jm} - y_{jk}y_{jm}) \\
   = (G^x_{km} - G^y_{km})(G^x_{km} - G^y_{km}) \\
   = \left\Vert G^x - G^y\right\Vert_F^2
  \end{array}
  $$</span></p>
  <p>The authors of the <a href="https://arxiv.org/abs/1701.01036" rel="nofollow noreferrer">1701.01036</a> also tried different kernel functions <span class="math-container">$k(x,y)$</span> getting more interesting style transfer results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-17T22:57:17.470Z
Answers.Text: Hi, thanks for the thorough reply, just need a couple of clarifications to allow me to fully understand this. The MMD^2 portion you've written is basically a squared L2 Norm applied to the feature vectors of X and Y that are unrolled and then subtracted, correct? Also, I am sorry if I am unfamiliar with common notation but what does the E represent?
Answers.UserDisplayName: Jaskeerat Singh Sarin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-20T07:21:09.177Z
Answers.Text: @JaskeeratSinghSarin The $\mathbb{E}$ represents "in expectation"- it assumes some population or probability distribution of observed variables. The basic equation you can bear in mind when converting to/from expectations is that if you have random variable $X$ that follows discrete probability distribution $p(x) = \mathbf{Pr}\{X=x\}$, then $\mathbb{E}[X] = \sum_{x} p(x)x$
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-17T12:21:11.267Z
Answers.LastActivityDate: 2021-05-17T20:38:01.853Z
Answers.LastEditDate: 2021-05-17T20:38:01.853Z
Answers.UserDisplayName: Kostya
Body: <p>I have been learning about Style Transfer recently. Style is defined as</p>
  <blockquote>
  <p>The correlation of activations between channels.</p>
  </blockquote>
  <p>I can't seem to understand why that would be true. Intuitively, style seems to be the patterns that exist in one particular channel/image rather than the patterns between channels. When filters in CNNs have different weights for acting as filters for different channels, why do we even expect 2 channels to be correlated? And further, why do we expect the style to be conveyed by it?</p>
  <p>I expected a style function that could compare activations in some layer of a CNN condensed into one channel so that an algorithm can search for which activations occur simultaneously and hold style information.</p>
  <p>I understand how we are carrying out the operations with the matrix and defining the loss function, what I don't get is why we are assuming style information lies in correlation between channels in the first place.</p>
  
Tags: <machine-learning><convolutional-neural-networks><generative-adversarial-networks><image-processing><style-transfer>
Title: In style transfer, why does the comparison between channels give a good sense of style?
UserDisplayName: Jaskeerat Singh Sarin
ThreadText: # In style transfer, why does the comparison between channels give a good sense of style?
  
  (Asked by: username_0 on 2021-05-17)
  
  I have been learning about Style Transfer recently. Style is defined as
  
  > The correlation of activations between channels.
  
  I can't seem to understand why that would be true. Intuitively, style seems to be the patterns that exist in one particular channel/image rather than the patterns between channels. When filters in CNNs have different weights for acting as filters for different channels, why do we even expect 2 channels to be correlated? And further, why do we expect the style to be conveyed by it?
  
  I expected a style function that could compare activations in some layer of a CNN condensed into one channel so that an algorithm can search for which activations occur simultaneously and hold style information.
  
  I understand how we are carrying out the operations with the matrix and defining the loss function, what I don't get is why we are assuming style information lies in correlation between channels in the first place.
  
  **Tags:** machine-learning, convolutional-neural-networks, generative-adversarial-networks, image-processing, style-transfer
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-17. Score: 2 | Accepted answer
  
  When the [original Neural Transfer paper](https://arxiv.org/abs/1508.06576) was published, your question stayed unanswered for a while. The reason why the [Gram matrix](https://en.wikipedia.org/wiki/Gram_matrix) represents artistic style was not entirely clear. A satisfactory (in my opinion) explanation came with the [*"Demystifying Neural Style Transfer"*](https://arxiv.org/abs/1701.01036) paper.
  
  The basic idea is that you cannot just directly compare activations for two images. The spatial positions of various features are different for the source and target image, so you should somehow get rid of positional information of the activations and compare their *distributions* across the whole image. The goal of the style transfer task is thus to make the two distributions as close as possible. One of the possible measures of distance between two distributions $P$ and $Q$ is the [Maximum Mean Distance](https://en.wikipedia.org/wiki/Kernel_embedding_of_distributions#Kernel_two-sample_test) (MMD):
  
  $$\text{MMD}^2(P,Q) = \left\Vert \mathbb{E}\_{P}[\phi(X)] - \mathbb{E}\_{Q}[\phi(Y)]\right\Vert^2$$
  
  With $\phi(\cdot)$ being a feature function - in our case it would be the NN activations in a particular layer. The next step would be to apply a "[kernel trick](https://en.wikipedia.org/wiki/Kernel_method)" to the MMD, representing it through a kernel $k(x,y)$
  
  $$\text{MMD}^2(P,Q) = \mathbb{E}[k(x\_i,x\_j)] + \mathbb{E}[k(y\_i,y\_j)] - 2 \mathbb{E}[k(x\_i,y\_j)]$$
  
  The Gram matrix of the original style transfer corresponds to the squared dot-product kernel $k(x,y) = (x^Ty)^2$:
  $$\text{MMD}^2(P,Q) = \mathbb{E}[(x^T\_ix\_j)^2] + \mathbb{E}[(y^T\_iy\_j)^2] - 2 \mathbb{E}[(x^T\_iy\_j)^2] = \left\Vert G^x - G^y\right\Vert\_F^2 $$
  
  Where $G^x$ and $G^y$ are the Gram matrices for activations and $\Vert\cdot\Vert\_F$ is the [Frobenius norm](https://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm). The last equality gives a bit of trouble to those not used to the kernel trick, so I'll expand it (assuming [Einstein summation](https://en.wikipedia.org/wiki/Einstein_notation) convention):
  
  $$\begin{array}{l}
  \mathbb{E}[(x^T\_ix\_j)^2] + \mathbb{E}[(y^T\_iy\_j)^2] - 2 \mathbb{E}[(x^T\_iy\_j)^2] = \\
  = x\_{ik}x\_{jk}x\_{im}x\_{jm} + y\_{ik}y\_{jk}y\_{im}y\_{jm} - 2 x\_{ik}x\_{jk}y\_{im}y\_{jm} \\
  = (x\_{ik}x\_{im} - y\_{ik}y\_{im})(x\_{jk}x\_{jm} - y\_{jk}y\_{jm}) \\
  = (G^x\_{km} - G^y\_{km})(G^x\_{km} - G^y\_{km}) \\
  = \left\Vert G^x - G^y\right\Vert\_F^2
  \end{array}
  $$
  
  The authors of the [1701.01036](https://arxiv.org/abs/1701.01036) also tried different kernel functions $k(x,y)$ getting more interesting style transfer results.
  
  > **Comments:**
  > **username_0**: Hi, thanks for the thorough reply, just need a couple of clarifications to allow me to fully understand this. The MMD^2 portion you've written is basically a squared L2 Norm applied to the feature vectors of X and Y that are unrolled and then subtracted, correct? Also, I am sorry if I am unfamiliar with common notation but what does the E represent?  
  > **username_2**: @username_0 The $\mathbb{E}$ represents "in expectation"- it assumes some population or probability distribution of observed variables. The basic equation you can bear in mind when converting to/from expectations is that if you have random variable $X$ that follows discrete probability distribution $p(x) = \mathbf{Pr}\{X=x\}$, then $\mathbb{E}[X] = \sum_{x} p(x)x$  
  
  ---
  
  