Answers.Body: <p>Both are incorrect.  </p>
  
  <p><em>using your notation</em><br>
  You do not take a sliding frobenius inner product of a singular channel of <span class="math-container">$I$</span> with <span class="math-container">$F$</span>, but with all the channels at once. This may be easier to understand if you do not assume the number of channels of the input and output are the same (ie different number of input channels then filters). So lets say your input has <span class="math-container">$k_1$</span> channels and you have <span class="math-container">$k_2$</span> filters.  </p>
  
  <p>This means <span class="math-container">$shape(I) = (N, M, k_1)$</span> and each of the <span class="math-container">$k_2$</span> filters is of shape <span class="math-container">$(n, m,k_2)$</span> and your output shape is <span class="math-container">$(N-n+1, M-m+1, k_2)$</span> assuming your not using padding.  </p>
  
  <p>So i guess trying to put it in the manner you use: The convolution's output's <span class="math-container">$i^{th}$</span> channel is taking the sliding Frobenius inner product between all <span class="math-container">$n \times m$</span> cross-section of <span class="math-container">$I$</span> (including <strong>all input channels</strong>) with filter <span class="math-container">$F_i$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T04:49:30.123Z
Answers.Text: "You do not take a sliding frobenius inner product of a singular channel of I with F, but with all the channels at once." That is what I intended to say with "computing the Frobenius inner product between channel $i$ and kernel $i$ at each position". Given that, what you are saying is the same as option 1, right? ONE convolution produces ONE output channel (feature map) $o$. Not ONE convolution produces $k$ matrices that are THEN then (in a spererate step, which is not part of the convolution operation) added component-wise to obtain $o$. Correct?
Answers.UserDisplayName: lo tolmencre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T06:12:43.433Z
Answers.Text: Just to make sure, we are on the same page: See http://yann.lecun.com/exdb/publis/pdf/jarrett-iccv-09.pdf for where they used the second definition. There they sum over the convolutions of one filter to get one output channel. So for them a convolution is between a channel (2D) and a kernel (2D).
Answers.UserDisplayName: lo tolmencre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T06:33:56.730Z
Answers.Text: I also added formal definitions to my question now.
Answers.UserDisplayName: lo tolmencre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T06:59:19.040Z
Answers.Text: In simpler terms, each filter in a convolutional layer multiplies through the entirety of the input depth. So for a filter of size 3x3 (stride 1) sliding over an image of 28x28x3, each output node (of which there will be (28-3)/1+1, 26x26, so 676) will be the sum of the filter multiplied through all 3 channels of the input at the filters location(put through some activation function)
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T07:05:24.800Z
Answers.Text: @Recessive I know. I want to know which of these operations is part of the convolution _definition_ and which part is an extra step not included in the convolution operation itself.
Answers.UserDisplayName: lo tolmencre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T07:14:39.483Z
Answers.Text: @lotolmencre Oh I see, sorry for misunderstanding. I think this is a tricky one to answer, but I would say Option 1, as all instances of a convolutional layer I have seen have referred to at most the act of convolving and passing the outputs through an activation function as separate segments, nevermind the act of convolving (but not summing), then summing, then passing the outputs through an activation function. The reason I say it is hard to answer is I don't believe there is a formal definition (in terms of machine learning)
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T07:18:06.043Z
Answers.Text: @lotolmencre From my understanding, the term "convolution" in regards to neural networks is a loose interpretation of the typical definition, which "is a mathematical operation on two functions (f and g) to produce a third function that expresses how the shape of one is modified by the other." I think this supports option 1, especially in the fact "The term convolution refers to both the result function and to the process of computing it." But even so, I can not answer this with 100% confidence, because I am unsure of a formal definition in regards to ML.
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T12:39:55.413Z
Answers.Text: (sorry for late reply, was asleep) Its is the **exact** same definition just in the 2 dimensional sense of discrete convolution. Regarding the link you sent: Are you talking about the average pooling at the end? if so, that is an additional layer/op seperate form the convolution.
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T12:42:28.663Z
Answers.Text: also note that after your edits: your definitions still are wrong, but the equation in option 1 is correct!
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T12:59:50.640Z
Answers.Text: @mshlis I am referring to equation (1) in the paragraph _Filter Bank Layer_ .  That is their definition of convolution and it is restricted to one channel per convolution. Where do you find my definition to diverge from the equation? I modeled the equation after my prose definition. So the problem must be in bad wording rather than misconception.
Answers.UserDisplayName: lo tolmencre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T13:01:06.333Z
Answers.Text: "between channel k and kernerl k at each position"
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T13:02:52.347Z
Answers.Text: @mshlis What is incorrect about that? That is the outer sum in the formula $\sum_{k=1}^d$.
Answers.UserDisplayName: lo tolmencre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T13:29:00.360Z
Answers.Text: that is doing all channels with kernel $k$, note that your sum goes over all of them, so your frobenius ip is with all channels
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T13:30:47.323Z
Answers.Text: Oh, ok. Yea well, the imprecise nature of prose. I meant $k$ as a running index in that formulation. Probably should have said "between channelS k and kernerlS k at each position".
Answers.UserDisplayName: lo tolmencre
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T13:33:21.180Z
Answers.Text: then i dont get why $k$ would refer to a set of channels, but yeah i think at this point you get what im saying and were now just dealing with semantics
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-25T01:02:40.330Z
Answers.LastActivityDate: 2019-07-25T01:02:40.330Z
Answers.UserDisplayName: mshlis
Body: <p>From the literature I have read so far, it is not clear how exactly the convolution operation is defined. It seems people use two different definitions:</p>
  <p>Let us assume we are given an <span class="math-container">$n_w \times n_h \times d$</span> input tensor <span class="math-container">$I$</span> and an <span class="math-container">$m_w \times m_h \times d$</span> filter <span class="math-container">$F$</span> of <span class="math-container">$d$</span> kernels (I use the convention of referring to the depth-slices of filters as <em>kernels</em>. I also will call the depth slices of the input tensor <em>channels</em>). Let us also assume <span class="math-container">$F$</span> is the <span class="math-container">$j$</span>th filter of <span class="math-container">$J$</span> filters.</p>
  <p>Now to the definitions.</p>
  <p><strong>Option 1:</strong></p>
  <p>The convolution of <span class="math-container">$I$</span> with <span class="math-container">$F$</span> is obtained by sliding <span class="math-container">$F$</span> across <span class="math-container">$I$</span> and computing the Frobenius inner product between channel <span class="math-container">$k$</span> and kernel <span class="math-container">$k$</span> at each position, adding the products and storing them in an output matrix. That matrix is the result of the convolution. It is also the <span class="math-container">$j$</span>th feature map in the output tensor of the convolution layer.</p>
  <p>Let <span class="math-container">$I \in \mathbb{R}^{n_w \times n_h \times d}$</span> and <span class="math-container">$F \in \mathbb{R}^{m_w \times m_h \times d}$</span>. Let <span class="math-container">$s \in \mathbb{N}$</span> be the stride. The operation will only be defined if the smaller tensor fits within the larger tensor along its width and height a positive integer number of times when shifting by <span class="math-container">$s$</span>, that is if and only if <span class="math-container">$k_w = (n_w - m_w) / s + 1\in \mathbb{N}$</span> and <span class="math-container">$k_h = (n_h - m_h) / s + 1\in \mathbb{N}$</span>, where <span class="math-container">$k_w \times k_h \times d$</span> is the shape of the output tensor. Furthermore let <span class="math-container">${f_x : i \mapsto (x - 1)s + i}$</span> be a function that returns the absolute index in the input tensor, given an index <span class="math-container">$x$</span> in the output tensor, the stride length <span class="math-container">$s$</span> and a relative index <span class="math-container">$i$</span>.
  <span class="math-container">\begin{equation*}
      \begin{split}
          (I * F)_{x y} =
              &amp; \sum_{k=1}^d \sum_{i = 1}^{m_w} \sum_{j = 1}^{m_h} I_{f_x(i) f_y(j) k} \cdot F_{i j k}
      \end{split}
  \end{equation*}</span></p>
  <p><strong>Option 2:</strong></p>
  <p>The convolutions (plural) of <span class="math-container">$I$</span> with <span class="math-container">$F$</span> are obtained by sliding <span class="math-container">$F$</span> across <span class="math-container">$I$</span> and computing the Frobenius inner product between channel <span class="math-container">$k$</span> and kernel <span class="math-container">$i$</span> at each position. Each product is stored in a matrix associated with the channel <span class="math-container">$k$</span>. There is no adding of the products yet. The convolutions are the result matrices. The step where the matrices are added component wise to obtain the <span class="math-container">$j$</span>th feature map of the output tensor of the convolution layer is not part of the convolution operation, but an independent step.</p>
  <p>Let <span class="math-container">$I \in \mathbb{R}^{n_w \times n_h}$</span> and <span class="math-container">$F \in \mathbb{R}^{m_w \times m_h}$</span>. Let <span class="math-container">$s \in \mathbb{N}$</span> be the stride. The operation will only be defined if the smaller matrix fits within the larger one along its width and height a positive integer number of times when shifting by <span class="math-container">$s$</span>, that is, if and only if <span class="math-container">$k_w = (n_w - m_w) / s + 1\in \mathbb{N}$</span> and <span class="math-container">$k_h = (n_h - m_h) / s + 1\in \mathbb{N}$</span>, where <span class="math-container">$k_w \times k_h$</span> is the shape of the output matrix. Furthermore let <span class="math-container">${f_x : i \mapsto (x - 1)s + i}$</span> be a function that returns the absolute index in the input matrix, given an index <span class="math-container">$x$</span> in the output matrix, the stride length <span class="math-container">$s$</span> and a relative index <span class="math-container">$i$</span>.
  <span class="math-container">\begin{equation*}
      \begin{split}
          (I * F)_{x y} =
              &amp;\sum_{i = 1}^{m_w} \sum_{j = 1}^{m_h} I_{f_x(i) f_y(j)} \cdot F_{i j}
      \end{split}
  \end{equation*}</span></p>
  <p>Which of these two definitions is the common one?</p>
  
Tags: <neural-networks><convolutional-neural-networks><definitions><convolution>
Title: Is adding the Frobenius inner products between filter and input part of convolution or a separate step?
UserDisplayName: lo tolmencre
ThreadText: # Is adding the Frobenius inner products between filter and input part of convolution or a separate step?
  
  (Asked by: username_0 on 2019-07-24)
  
  From the literature I have read so far, it is not clear how exactly the convolution operation is defined. It seems people use two different definitions:
  
  Let us assume we are given an $n\_w \times n\_h \times d$ input tensor $I$ and an $m\_w \times m\_h \times d$ filter $F$ of $d$ kernels (I use the convention of referring to the depth-slices of filters as *kernels*. I also will call the depth slices of the input tensor *channels*). Let us also assume $F$ is the $j$th filter of $J$ filters.
  
  Now to the definitions.
  
  **Option 1:**
  
  The convolution of $I$ with $F$ is obtained by sliding $F$ across $I$ and computing the Frobenius inner product between channel $k$ and kernel $k$ at each position, adding the products and storing them in an output matrix. That matrix is the result of the convolution. It is also the $j$th feature map in the output tensor of the convolution layer.
  
  Let $I \in \mathbb{R}^{n\_w \times n\_h \times d}$ and $F \in \mathbb{R}^{m\_w \times m\_h \times d}$. Let $s \in \mathbb{N}$ be the stride. The operation will only be defined if the smaller tensor fits within the larger tensor along its width and height a positive integer number of times when shifting by $s$, that is if and only if $k\_w = (n\_w - m\_w) / s + 1\in \mathbb{N}$ and $k\_h = (n\_h - m\_h) / s + 1\in \mathbb{N}$, where $k\_w \times k\_h \times d$ is the shape of the output tensor. Furthermore let ${f\_x : i \mapsto (x - 1)s + i}$ be a function that returns the absolute index in the input tensor, given an index $x$ in the output tensor, the stride length $s$ and a relative index $i$.
  \begin{equation\*}
  \begin{split}
  (I \* F)\_{x y} =
  & \sum\_{k=1}^d \sum\_{i = 1}^{m\_w} \sum\_{j = 1}^{m\_h} I\_{f\_x(i) f\_y(j) k} \cdot F\_{i j k}
  \end{split}
  \end{equation\*}
  
  **Option 2:**
  
  The convolutions (plural) of $I$ with $F$ are obtained by sliding $F$ across $I$ and computing the Frobenius inner product between channel $k$ and kernel $i$ at each position. Each product is stored in a matrix associated with the channel $k$. There is no adding of the products yet. The convolutions are the result matrices. The step where the matrices are added component wise to obtain the $j$th feature map of the output tensor of the convolution layer is not part of the convolution operation, but an independent step.
  
  Let $I \in \mathbb{R}^{n\_w \times n\_h}$ and $F \in \mathbb{R}^{m\_w \times m\_h}$. Let $s \in \mathbb{N}$ be the stride. The operation will only be defined if the smaller matrix fits within the larger one along its width and height a positive integer number of times when shifting by $s$, that is, if and only if $k\_w = (n\_w - m\_w) / s + 1\in \mathbb{N}$ and $k\_h = (n\_h - m\_h) / s + 1\in \mathbb{N}$, where $k\_w \times k\_h$ is the shape of the output matrix. Furthermore let ${f\_x : i \mapsto (x - 1)s + i}$ be a function that returns the absolute index in the input matrix, given an index $x$ in the output matrix, the stride length $s$ and a relative index $i$.
  \begin{equation\*}
  \begin{split}
  (I \* F)\_{x y} =
  &\sum\_{i = 1}^{m\_w} \sum\_{j = 1}^{m\_h} I\_{f\_x(i) f\_y(j)} \cdot F\_{i j}
  \end{split}
  \end{equation\*}
  
  Which of these two definitions is the common one?
  
  **Tags:** neural-networks, convolutional-neural-networks, definitions, convolution
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-25. Score: 2 | Accepted answer
  
  Both are incorrect.
  
  *using your notation*  
  You do not take a sliding frobenius inner product of a singular channel of $I$ with $F$, but with all the channels at once. This may be easier to understand if you do not assume the number of channels of the input and output are the same (ie different number of input channels then filters). So lets say your input has $k\_1$ channels and you have $k\_2$ filters.
  
  This means $shape(I) = (N, M, k\_1)$ and each of the $k\_2$ filters is of shape $(n, m,k\_2)$ and your output shape is $(N-n+1, M-m+1, k\_2)$ assuming your not using padding.
  
  So i guess trying to put it in the manner you use: The convolution's output's $i^{th}$ channel is taking the sliding Frobenius inner product between all $n \times m$ cross-section of $I$ (including **all input channels**) with filter $F\_i$.
  
  > **Comments:**
  > **username_0**: "You do not take a sliding frobenius inner product of a singular channel of I with F, but with all the channels at once." That is what I intended to say with "computing the Frobenius inner product between channel $i$ and kernel $i$ at each position". Given that, what you are saying is the same as option 1, right? ONE convolution produces ONE output channel (feature map) $o$. Not ONE convolution produces $k$ matrices that are THEN then (in a spererate step, which is not part of the convolution operation) added component-wise to obtain $o$. Correct?  
  > **username_0**: Just to make sure, we are on the same page: See http://yann.lecun.com/exdb/publis/pdf/jarrett-iccv-09.pdf for where they used the second definition. There they sum over the convolutions of one filter to get one output channel. So for them a convolution is between a channel (2D) and a kernel (2D).  
  > **username_0**: I also added formal definitions to my question now.  
  > **username_2**: In simpler terms, each filter in a convolutional layer multiplies through the entirety of the input depth. So for a filter of size 3x3 (stride 1) sliding over an image of 28x28x3, each output node (of which there will be (28-3)/1+1, 26x26, so 676) will be the sum of the filter multiplied through all 3 channels of the input at the filters location(put through some activation function)  
  > **username_0**: @username_2 I know. I want to know which of these operations is part of the convolution _definition_ and which part is an extra step not included in the convolution operation itself.  
  > **username_2**: @username_0 Oh I see, sorry for misunderstanding. I think this is a tricky one to answer, but I would say Option 1, as all instances of a convolutional layer I have seen have referred to at most the act of convolving and passing the outputs through an activation function as separate segments, nevermind the act of convolving (but not summing), then summing, then passing the outputs through an activation function. The reason I say it is hard to answer is I don't believe there is a formal definition (in terms of machine learning)  
  > **username_2**: @username_0 From my understanding, the term "convolution" in regards to neural networks is a loose interpretation of the typical definition, which "is a mathematical operation on two functions (f and g) to produce a third function that expresses how the shape of one is modified by the other." I think this supports option 1, especially in the fact "The term convolution refers to both the result function and to the process of computing it." But even so, I can not answer this with 100% confidence, because I am unsure of a formal definition in regards to ML.  
  > **username_1**: (sorry for late reply, was asleep) Its is the **exact** same definition just in the 2 dimensional sense of discrete convolution. Regarding the link you sent: Are you talking about the average pooling at the end? if so, that is an additional layer/op seperate form the convolution.  
  > **username_1**: also note that after your edits: your definitions still are wrong, but the equation in option 1 is correct!  
  > **username_0**: @username_1 I am referring to equation (1) in the paragraph _Filter Bank Layer_ .  That is their definition of convolution and it is restricted to one channel per convolution. Where do you find my definition to diverge from the equation? I modeled the equation after my prose definition. So the problem must be in bad wording rather than misconception.  
  > **username_1**: "between channel k and kernerl k at each position"  
  > **username_0**: @username_1 What is incorrect about that? That is the outer sum in the formula $\sum_{k=1}^d$.  
  > **username_1**: that is doing all channels with kernel $k$, note that your sum goes over all of them, so your frobenius ip is with all channels  
  > **username_0**: Oh, ok. Yea well, the imprecise nature of prose. I meant $k$ as a running index in that formulation. Probably should have said "between channelS k and kernerlS k at each position".  
  > **username_1**: then i dont get why $k$ would refer to a set of channels, but yeah i think at this point you get what im saying and were now just dealing with semantics  
  
  ---
  
  