Answers.Body: <p>First for clarity I want to differ &quot;convolutional kernel&quot; and &quot;filter&quot; here, let's say a filter has several convolutional kernels. Second in NLP 1D-convolution is mostly used, so different from CV, which typically use a <span class="math-container">$k \times k$</span> convolutional kernel, in NLP generally the kernel size is only <span class="math-container">$k$</span></p>
  <p>Having listed above hope things could become a little clearer: If we adopt the same concepts as we seen in typical CV scenario, per my understanding, the size of convolutional kernel is only <span class="math-container">$k$</span>, but the count of kernels, or the dimension of new &quot;channel&quot;, is set to <span class="math-container">$2d$</span>. One kernel maps <span class="math-container">$k \times d$</span> elements into one element, <span class="math-container">$2d$</span> kernels together convert the input to a vector of which the dimention is <span class="math-container">$2d$</span></p>
  <p>As the calculation process is mapping a matrix <span class="math-container">$\mathbb{R}^{k \times d}$</span> to a vector <span class="math-container">$\mathbb{R}^{2d \times 1}$</span>, if expressed in a form of matrix multiplication, or in a full-connected view, we can firstly expand the input matrix, concatenate all rows together to get a new big vector of which the dimension is <span class="math-container">$\mathbb{R}^{kd \times 1}$</span>, then multiply a weight matrix <span class="math-container">$\boldsymbol{W} \in \mathbb{R}^{2d \times kd}$</span> could get the final vector</p>
  <p>The key problem that roots the confusion is beause, in my opinion, the &quot;kernel&quot; imported here is a little different from the same concept used in a typical CNN. Again, if we describe in a traditional CNN terminology, the kernel size should be <span class="math-container">$k$</span>, and there are <span class="math-container">$2d$</span> kernels applied.</p>
  <p>Hope my understanding is correct.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-15T16:29:00.107Z
Answers.LastActivityDate: 2020-12-15T16:29:00.107Z
Answers.UserDisplayName: TX Shi
Answers.Body: <p>They are doing a matrix multiplication: consider <span class="math-container">$y = Ax, y \in \mathbb{R}^m, x \in \mathbb{R}^n, A \in M_\mathbb{R}(m,n)$</span>. In the paper <span class="math-container">$x$</span> is a concatenation of <span class="math-container">$k$</span> elements of <span class="math-container">$\mathbb{R}^d$</span>, so <span class="math-container">$x$</span> is long <span class="math-container">$kd$</span>; <span class="math-container">$y$</span> is long <span class="math-container">$2d$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-10T14:00:36.113Z
Answers.LastActivityDate: 2019-05-10T14:00:36.113Z
Answers.UserDisplayName: dcolazin
Body: <p>I am reading the paper Convolutional Sequence to Sequence Learning by Facebook AI researchers and having trouble to understand how the dimensions of convolutional filters work here. Please take a look at the relevant part of the paper below. </p>
  
  <p><a href="https://i.sstatic.net/1xSku.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1xSku.png" alt="enter image description here"></a></p>
  
  <p>Let's say the input to the kernel X is k*d (say k=5 words of d=300 embedding dimenisonality). Therefore the input is 5*300. In a computer vision task a kernel would slide over parts of the image, in NLP you usually see kernel taking up the whole width of the input matrix. So I would expect kernel to be m*d (e.g. 3*300 - slide over 3 words and look at their whole embeddiings).</p>
  
  <p>However, the kernel here is of dimensionality 2d x kd which in our hypothetical example would be 600*1500. I don't understand how this massive kernel would slide over an input that is by far lower dimensional (5*300). In computer vision you could zero-pad the input, but here zero-padding would basically turn the input matrix into mostly zeros with only a handful of meaningful numbers.</p>
  
  <p>Thanks for shedding some light on it!</p>
  
Tags: <convolutional-neural-networks><natural-language-processing><machine-translation>
Title: Convolutional Sequence to Sequence Learning kernel parameters
UserDisplayName: ytolochko
ThreadText: # Convolutional Sequence to Sequence Learning kernel parameters
  
  (Asked by: username_0 on 2019-05-10)
  
  I am reading the paper Convolutional Sequence to Sequence Learning by Facebook AI researchers and having trouble to understand how the dimensions of convolutional filters work here. Please take a look at the relevant part of the paper below.
  
  [![enter image description here](https://i.sstatic.net/1xSku.png)](https://i.sstatic.net/1xSku.png)
  
  Let's say the input to the kernel X is k\*d (say k=5 words of d=300 embedding dimenisonality). Therefore the input is 5\*300. In a computer vision task a kernel would slide over parts of the image, in NLP you usually see kernel taking up the whole width of the input matrix. So I would expect kernel to be m\*d (e.g. 3\*300 - slide over 3 words and look at their whole embeddiings).
  
  However, the kernel here is of dimensionality 2d x kd which in our hypothetical example would be 600\*1500. I don't understand how this massive kernel would slide over an input that is by far lower dimensional (5\*300). In computer vision you could zero-pad the input, but here zero-padding would basically turn the input matrix into mostly zeros with only a handful of meaningful numbers.
  
  Thanks for shedding some light on it!
  
  **Tags:** convolutional-neural-networks, natural-language-processing, machine-translation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-12-15. Score: 0
  
  First for clarity I want to differ "convolutional kernel" and "filter" here, let's say a filter has several convolutional kernels. Second in NLP 1D-convolution is mostly used, so different from CV, which typically use a $k \times k$ convolutional kernel, in NLP generally the kernel size is only $k$
  
  Having listed above hope things could become a little clearer: If we adopt the same concepts as we seen in typical CV scenario, per my understanding, the size of convolutional kernel is only $k$, but the count of kernels, or the dimension of new "channel", is set to $2d$. One kernel maps $k \times d$ elements into one element, $2d$ kernels together convert the input to a vector of which the dimention is $2d$
  
  As the calculation process is mapping a matrix $\mathbb{R}^{k \times d}$ to a vector $\mathbb{R}^{2d \times 1}$, if expressed in a form of matrix multiplication, or in a full-connected view, we can firstly expand the input matrix, concatenate all rows together to get a new big vector of which the dimension is $\mathbb{R}^{kd \times 1}$, then multiply a weight matrix $\boldsymbol{W} \in \mathbb{R}^{2d \times kd}$ could get the final vector
  
  The key problem that roots the confusion is beause, in my opinion, the "kernel" imported here is a little different from the same concept used in a typical CNN. Again, if we describe in a traditional CNN terminology, the kernel size should be $k$, and there are $2d$ kernels applied.
  
  Hope my understanding is correct.
  
  
  ---
  
  ### Answer by username_2 on 2019-05-10. Score: 0
  
  They are doing a matrix multiplication: consider $y = Ax, y \in \mathbb{R}^m, x \in \mathbb{R}^n, A \in M\_\mathbb{R}(m,n)$. In the paper $x$ is a concatenation of $k$ elements of $\mathbb{R}^d$, so $x$ is long $kd$; $y$ is long $2d$.
  
  
  ---
  
  