Answers.Body: <blockquote>
    <ol>
    <li><p>For the line <code>featRaw = selectedLayer.output</code>, when I print <code>featRaw</code>, I get the output: <code>Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32)</code>.</p>
    
    <ul>
    <li>a) <code>Relu:0</code> does this mean Relu activation has not yet been applied?</li>
    </ul></li>
    </ol>
  </blockquote>
  
  <p>It has been applied.</p>
  
  <blockquote>
    <ul>
    <li>b) Also I presume we're outputing the feature maps outputs from <code>block4_conv2</code>, not the filters/kernels themselves, correct?</li>
    </ul>
  </blockquote>
  
  <p>Yes.</p>
  
  <blockquote>
    <ul>
    <li>c) Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.</li>
    </ul>
  </blockquote>
  
  <p>The first dimension in keras is used for representing <code>batch</code> dimension. When training the model, you aren't passing the images one by one but in batches. The number of images that are processed in parallel before the model calculates its loss and makes its update is called the <code>batch_size</code>. When computing the content loss, though, you have a single image you want to perform style transfer to, so <code>batch_size=1</code>. That's why you see the unit in the beginning of the tensor's shape.</p>
  
  <blockquote>
    <ul>
    <li>d) Is <code>selectedLayer.output</code> simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?</li>
    </ul>
  </blockquote>
  
  <p>It outputs the tensor containing all the pixel values that are outputted from that layer.</p>
  
  <blockquote>
    <ol start="2">
    <li><p>With the line: <code>featMatrix = K.reshape(featRaw, (M_l, N_l)</code> where printing <code>featMatrix</code> would output: <code>Tensor("Reshape:0", shape=(4096, 512), dtype=float32)</code>.</p>
    
    <ul>
    <li>a) This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!</li>
    </ul></li>
    </ol>
  </blockquote>
  
  <p>I think you're confusing things a bit. You <strong>are</strong> getting the pixel values, just in a <strong>different shape</strong>. Instead of stacking them in a <span class="math-container">$64 \times 64 \times 515$</span> array you are stacking them in a <span class="math-container">$1096 \times 515$</span> array. <strong>Exactly</strong> the same information is stored in a different shape. Think of it like transforming a <span class="math-container">$10 \times 2$</span> matrix to a <span class="math-container">$5 \times 4$</span> one. It contains the same information, in a different shape.</p>
  
  <blockquote>
    <ul>
    <li>b) Also the final <code>featMatrix</code> is transposed - i.e. <code>featMatrix = K.transpose(featMatrix)</code> with the output <code>Tensor("transpose:0", shape=(512, 4096), dtype=float32)</code>. Why is that (i.e. why reverse the axis)?</li>
    </ul>
  </blockquote>
  
  <p>I'm not sure why exactly he does this, but from a bit that I saw the code I think it has something to do with the style loss (not the content loss). Probably something didn't match up with the dimensions of some operation, maybe a matrix multiplication...</p>
  
  <blockquote>
    <ol start="3">
    <li>Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?</li>
    </ol>
  </blockquote>
  
  <p>If you have the output image as a tensor, let's say <code>y</code>. You can convert it to numpy through <a href="https://keras.io/backend/#eval" rel="nofollow noreferrer"><code>keras.backend.eval</code></a>:</p>
  
  
  
  <pre><code>import keras.backend as K
  
  y_arr = K.eval(y)
  </code></pre>
  
  <p>There are many ways to save the image, including <a href="https://scikit-image.org/docs/dev/api/skimage.io.html#skimage.io.imsave" rel="nofollow noreferrer">scikit-image</a>, <a href="https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.misc.imsave.html" rel="nofollow noreferrer">scipy</a>, <a href="https://docs.opencv.org/2.4/modules/highgui/doc/reading_and_writing_images_and_video.html#imwrite" rel="nofollow noreferrer">opencv</a>, <a href="https://pillow.readthedocs.io/en/3.1.x/reference/Image.html#PIL.Image.Image.save" rel="nofollow noreferrer">PIL</a>, <a href="https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.imsave.html" rel="nofollow noreferrer">matplotlib</a>, etc. I'll use the last:</p>
  
  
  
  <pre><code>import matplotlib.pyplot as plt
  
  plt.imsave('my_image.png', y_arr)
  </code></pre>
  
  <p>Note that in order to work <code>y_arr</code> should have a shape of either <code>(height, width, 3)</code> (if it is an RGB image) or <code>(height, width)</code> (if it is grayscale).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-30T23:10:25.937Z
Answers.Text: Thank you so much for the response and clarification. Much appreciated. 
  I did the `K.eval(P)` and it worked. `P` is the output content representation tensor: `P = get_feature_reps(x=cImArr, layer_names=[cLayerName], model=cModel)[0]`. Using `K.eval(P)` I got an output dtype of `float32` and a shape of `(512, 4096)`. So just referring to your very last answer, for ensuring to convert to the correct format for RGB image, how do I convert a shape of `(512, 4096)` to the correct shape of `(height, width, 3)`.
Answers.UserDisplayName: Hazzaldo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-30T23:10:39.007Z
Answers.Text: When I saved the array to an image with the shape `(512, 4096)`, the image looked completely weird: https://imgur.com/XHUlqtY  It looks nothing like the original image: https://imgur.com/oZrBlX0
Answers.UserDisplayName: Hazzaldo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-30T23:11:16.803Z
Answers.Text: My second follow up question is, the `eval()` function did not work on the style representation tensor data structure (`As`), for this code output: `As = get_feature_reps(x=sImArr, layer_names=sLayerNames, model=sModel)`.
Answers.UserDisplayName: Hazzaldo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-30T23:11:44.607Z
Answers.Text: When I tried `K.eval(As)` I got the error message: `'list' object has no attribute 'eval'`.  When I print out `As` I get the output: `[<tf.Tensor 'transpose_18:0' shape=(64, 262144) dtype=float32>,
   <tf.Tensor 'transpose_19:0' shape=(128, 65536) dtype=float32>,
   <tf.Tensor 'transpose_20:0' shape=(256, 16384) dtype=float32>,
   <tf.Tensor 'transpose_21:0' shape=(512, 4096) dtype=float32>]`
Answers.UserDisplayName: Hazzaldo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-30T23:12:24.973Z
Answers.Text: Sorry about the follow up questions.
Answers.UserDisplayName: Hazzaldo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-31T23:38:50.447Z
Answers.Text: You first need to transpose the array to `(4096, 512)`. You can do this by `y_arr = y_arr.T`. Then you need to reshape it to the original dimensions: `y_arr = y_arr.reshape((64, 64, 512))`. I don't think you can turn these into rgb images. I think this represents 512 grayscale feature maps. To get each you can simply slice the array, e.g. `y_arr[:, :, 13]` (for the image with index 13).
Answers.UserDisplayName: Djib2011
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-31T23:42:54.123Z
Answers.Text: You can think of `K.eval()` as a method of converting tensors to arrays. It can't be applied to `As` because `As` is a list not a tensor. Instead it can be applied to the contents of the list, which happen to be tensors, e.g. `K.eval(As[0])` for the first tensor from the list.
Answers.UserDisplayName: Djib2011
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T11:37:05.320Z
Answers.Text: Thank you so much for your help :)  Just a side note btw, `y_arr.T` did not work, as it was throwing an error: `'Tensor' object has no attribute 'T'`. I tried `tf.transpose(y_arr)` and that worked. Thanks again. Really appreciate your help on this. :)
Answers.UserDisplayName: Hazzaldo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T13:30:17.347Z
Answers.Text: `y_arr` is the array that came out from `K.eval(y)`. It should be an array not a tensor.
Answers.UserDisplayName: Djib2011
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T13:30:40.273Z
Answers.Text: You're welcome :)
Answers.UserDisplayName: Djib2011
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T13:39:05.580Z
Answers.Text: Oh ye of course. That makes sense now. It needed to be converted from Tensor to numpy array first to work with `.T`. Silly error on my part :)  Thanks again.
Answers.UserDisplayName: Hazzaldo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-29T19:20:15.600Z
Answers.LastActivityDate: 2019-08-29T19:20:15.600Z
Answers.UserDisplayName: Djib2011
Body: <p>I'm trying to understand the intuition behind how the Content Loss is calculated in a Neural Style Transfer. I'm reading from an articles: <a href="https://medium.com/mlreview/making-ai-art-with-style-transfer-using-keras-8bb5fa44b216" rel="nofollow noreferrer">https://medium.com/mlreview/making-ai-art-with-style-transfer-using-keras-8bb5fa44b216</a> ,  that explains the implementation of Neural Style Transfer, from the Content loss function:
  <a href="https://i.sstatic.net/MiyOW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MiyOW.png" alt="enter image description here"></a></p>
  
  <p>The article explains that: </p>
  
  <ul>
  <li><p>F and P are matrices with a number of rows equal to N and a number of columns equal to M.</p></li>
  <li><p>N is the number of filters in layer l and M is the number of spatial elements in the feature map (height times width) for layer l.</p></li>
  </ul>
  
  <p>From the code below for getting the features/content representation from particular Conv layers, I didn't quite understand how it works. Basically I printed out the output of every line of code to try to make it easier, but it still left a number of questions to be asked, which I listed below the code: </p>
  
  <pre><code>def get_feature_reps(x, layer_names, model):
      """
      Get feature representations of input x for one or more layers in a given model.
      """
      featMatrices = []
      for ln in layer_names:
          selectedLayer = model.get_layer(ln)
          featRaw = selectedLayer.output
          featRawShape = K.shape(featRaw).eval(session=tf_session)
          N_l = featRawShape[-1]
          M_l = featRawShape[1]*featRawShape[2]
          featMatrix = K.reshape(featRaw, (M_l, N_l))
          featMatrix = K.transpose(featMatrix)
          featMatrices.append(featMatrix)
      return featMatrices
  
  def get_content_loss(F, P):
      cLoss = 0.5*K.sum(K.square(F - P))
      return cLoss
  </code></pre>
  
  <p>1- For the line <code>featRaw = selectedLayer.output</code>, when I print <code>featRaw</code>, I get the output:
  <code>Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32)</code>.</p>
  
  <ul>
  <li><p>a- <code>Relu:0</code> does this mean Relu activation has not yet been applied?</p></li>
  <li><p>b- Also I presume we're outputing the feature maps outputs from <code>block4_conv2</code>, not the filters/kernels themselves, correct?</p></li>
  <li><p>c- Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.</p></li>
  <li><p>d- Is <code>selectedLayer.output</code> simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?</p></li>
  </ul>
  
  <p>2- With the line: <code>featMatrix = K.reshape(featRaw, (M_l, N_l)</code> where printing <code>featMatrix</code> would output: <code>Tensor("Reshape:0", shape=(4096, 512), dtype=float32)</code>. </p>
  
  <ul>
  <li><p>a- This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!! </p></li>
  <li><p>b- Also the final <code>featMatrix</code> is transposed - i.e. <code>featMatrix = K.transpose(featMatrix)</code> with the output <code>Tensor("transpose:0", shape=(512, 4096), dtype=float32)</code>. Why is that (i.e. why reverse the axis)?</p></li>
  </ul>
  
  <p>3 - Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image? </p>
  
  <p>Any help would be really appreciated.</p>
  
Tags: <neural-networks><machine-learning><convolutional-neural-networks><tensorflow><keras>
Title: Understanding the intuition behind Content Loss (Neural Style Transfer)
UserDisplayName: Hazzaldo
ThreadText: # Understanding the intuition behind Content Loss (Neural Style Transfer)
  
  (Asked by: username_0 on 2019-08-28)
  
  I'm trying to understand the intuition behind how the Content Loss is calculated in a Neural Style Transfer. I'm reading from an articles: <https://medium.com/mlreview/making-ai-art-with-style-transfer-using-keras-8bb5fa44b216> , that explains the implementation of Neural Style Transfer, from the Content loss function:
  [![enter image description here](https://i.sstatic.net/MiyOW.png)](https://i.sstatic.net/MiyOW.png)
  
  The article explains that:
  
  * F and P are matrices with a number of rows equal to N and a number of columns equal to M.
  * N is the number of filters in layer l and M is the number of spatial elements in the feature map (height times width) for layer l.
  
  From the code below for getting the features/content representation from particular Conv layers, I didn't quite understand how it works. Basically I printed out the output of every line of code to try to make it easier, but it still left a number of questions to be asked, which I listed below the code:
  
  ```
  def get_feature_reps(x, layer_names, model):
      """
      Get feature representations of input x for one or more layers in a given model.
      """
      featMatrices = []
      for ln in layer_names:
          selectedLayer = model.get_layer(ln)
          featRaw = selectedLayer.output
          featRawShape = K.shape(featRaw).eval(session=tf_session)
          N_l = featRawShape[-1]
          M_l = featRawShape[1]*featRawShape[2]
          featMatrix = K.reshape(featRaw, (M_l, N_l))
          featMatrix = K.transpose(featMatrix)
          featMatrices.append(featMatrix)
      return featMatrices
  
  def get_content_loss(F, P):
      cLoss = 0.5*K.sum(K.square(F - P))
      return cLoss
  
  ```
  
  1- For the line `featRaw = selectedLayer.output`, when I print `featRaw`, I get the output:
  `Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32)`.
  
  * a- `Relu:0` does this mean Relu activation has not yet been applied?
  * b- Also I presume we're outputing the feature maps outputs from `block4_conv2`, not the filters/kernels themselves, correct?
  * c- Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.
  * d- Is `selectedLayer.output` simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?
  
  2- With the line: `featMatrix = K.reshape(featRaw, (M_l, N_l)` where printing `featMatrix` would output: `Tensor("Reshape:0", shape=(4096, 512), dtype=float32)`.
  
  * a- This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height \* width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!
  * b- Also the final `featMatrix` is transposed - i.e. `featMatrix = K.transpose(featMatrix)` with the output `Tensor("transpose:0", shape=(512, 4096), dtype=float32)`. Why is that (i.e. why reverse the axis)?
  
  3 - Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?
  
  Any help would be really appreciated.
  
  **Tags:** neural-networks, machine-learning, convolutional-neural-networks, tensorflow, keras
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-29. Score: 1 | Accepted answer
  
  > 1. For the line `featRaw = selectedLayer.output`, when I print `featRaw`, I get the output: `Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32)`.
  >
  >    * a) `Relu:0` does this mean Relu activation has not yet been applied?
  
  It has been applied.
  
  > * b) Also I presume we're outputing the feature maps outputs from `block4_conv2`, not the filters/kernels themselves, correct?
  
  Yes.
  
  > * c) Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.
  
  The first dimension in keras is used for representing `batch` dimension. When training the model, you aren't passing the images one by one but in batches. The number of images that are processed in parallel before the model calculates its loss and makes its update is called the `batch_size`. When computing the content loss, though, you have a single image you want to perform style transfer to, so `batch_size=1`. That's why you see the unit in the beginning of the tensor's shape.
  
  > * d) Is `selectedLayer.output` simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?
  
  It outputs the tensor containing all the pixel values that are outputted from that layer.
  
  > 2. With the line: `featMatrix = K.reshape(featRaw, (M_l, N_l)` where printing `featMatrix` would output: `Tensor("Reshape:0", shape=(4096, 512), dtype=float32)`.
  >
  >    * a) This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height \* width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!
  
  I think you're confusing things a bit. You **are** getting the pixel values, just in a **different shape**. Instead of stacking them in a $64 \times 64 \times 515$ array you are stacking them in a $1096 \times 515$ array. **Exactly** the same information is stored in a different shape. Think of it like transforming a $10 \times 2$ matrix to a $5 \times 4$ one. It contains the same information, in a different shape.
  
  > * b) Also the final `featMatrix` is transposed - i.e. `featMatrix = K.transpose(featMatrix)` with the output `Tensor("transpose:0", shape=(512, 4096), dtype=float32)`. Why is that (i.e. why reverse the axis)?
  
  I'm not sure why exactly he does this, but from a bit that I saw the code I think it has something to do with the style loss (not the content loss). Probably something didn't match up with the dimensions of some operation, maybe a matrix multiplication...
  
  > 3. Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?
  
  If you have the output image as a tensor, let's say `y`. You can convert it to numpy through [`keras.backend.eval`](https://keras.io/backend/#eval):
  
  ```
  import keras.backend as K
  
  y_arr = K.eval(y)
  
  ```
  
  There are many ways to save the image, including [scikit-image](https://scikit-image.org/docs/dev/api/skimage.io.html#skimage.io.imsave), [scipy](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.misc.imsave.html), [opencv](https://docs.opencv.org/2.4/modules/highgui/doc/reading_and_writing_images_and_video.html#imwrite), [PIL](https://pillow.readthedocs.io/en/3.1.x/reference/Image.html#PIL.Image.Image.save), [matplotlib](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.imsave.html), etc. I'll use the last:
  
  ```
  import matplotlib.pyplot as plt
  
  plt.imsave('my_image.png', y_arr)
  
  ```
  
  Note that in order to work `y_arr` should have a shape of either `(height, width, 3)` (if it is an RGB image) or `(height, width)` (if it is grayscale).
  
  > **Comments:**
  > **username_0**: Thank you so much for the response and clarification. Much appreciated.  I did the `K.eval(P)` and it worked. `P` is the output content representation tensor: `P = get_feature_reps(x=cImArr, layer_names=[cLayerName], model=cModel)[0]`. Using `K.eval(P)` I got an output dtype of `float32` and a shape of `(512, 4096)`. So just referring to your very last answer, for ensuring to convert to the correct format for RGB image, how do I convert a shape of `(512, 4096)` to the correct shape of `(height, width, 3)`.  
  > **username_0**: When I saved the array to an image with the shape `(512, 4096)`, the image looked completely weird: https://imgur.com/XHUlqtY  It looks nothing like the original image: https://imgur.com/oZrBlX0  
  > **username_0**: My second follow up question is, the `eval()` function did not work on the style representation tensor data structure (`As`), for this code output: `As = get_feature_reps(x=sImArr, layer_names=sLayerNames, model=sModel)`.  
  > **username_0**: When I tried `K.eval(As)` I got the error message: `'list' object has no attribute 'eval'`.  When I print out `As` I get the output: `[<tf.Tensor 'transpose_18:0' shape=(64, 262144) dtype=float32>,  <tf.Tensor 'transpose_19:0' shape=(128, 65536) dtype=float32>,  <tf.Tensor 'transpose_20:0' shape=(256, 16384) dtype=float32>,  <tf.Tensor 'transpose_21:0' shape=(512, 4096) dtype=float32>]`  
  > **username_0**: Sorry about the follow up questions.  
  > **username_1**: You first need to transpose the array to `(4096, 512)`. You can do this by `y_arr = y_arr.T`. Then you need to reshape it to the original dimensions: `y_arr = y_arr.reshape((64, 64, 512))`. I don't think you can turn these into rgb images. I think this represents 512 grayscale feature maps. To get each you can simply slice the array, e.g. `y_arr[:, :, 13]` (for the image with index 13).  
  > **username_1**: You can think of `K.eval()` as a method of converting tensors to arrays. It can't be applied to `As` because `As` is a list not a tensor. Instead it can be applied to the contents of the list, which happen to be tensors, e.g. `K.eval(As[0])` for the first tensor from the list.  
  > **username_0**: Thank you so much for your help :)  Just a side note btw, `y_arr.T` did not work, as it was throwing an error: `'Tensor' object has no attribute 'T'`. I tried `tf.transpose(y_arr)` and that worked. Thanks again. Really appreciate your help on this. :)  
  > **username_1**: `y_arr` is the array that came out from `K.eval(y)`. It should be an array not a tensor.  
  > **username_1**: You're welcome :)  
  > **username_0**: Oh ye of course. That makes sense now. It needed to be converted from Tensor to numpy array first to work with `.T`. Silly error on my part :)  Thanks again.  
  
  ---
  
  