Answers.Body: <p>In an image you are pooling usually over some (n x n) set of positions which lets you maintain spatial correlation but on the other hand most 1D CNNs used for language modeling pool over the temporal axis completely annihilating all form of temporal correlation within the resulting feature vector. I will take it this difference is what confuses you.</p>
  
  <p>For simplicity sake Ill keep my explanation to max-pooling but it can extend to any of the others. </p>
  
  <p>MaxPooling as the name suggests just takes a maximum value from a set of candidates. In images you commonly see maxpooling over neighborhoods of pixels. This is because if we assume high activation is correlated to some feature being more prevalent (this assumption is not inherent, its more learned than anything else by construction of the architecture/training), then doing this we keep the most prevalent features, maintain spatial correlation, and reduce the dimensionality.</p>
  
  <p>When using 1D CNNs for LMs rather than RNNs the idea is that each kernel is <em>searching</em> for a feature, and so you use an exorbitant number of the kernels at various sizes to <em>search</em> for a lot of features. Pooling over the temporal dimension then just tells you at each neuron how prevalent its corresponding feature is in that time series input. That is why your output is usually a feature vector of size equal to the number of kernels you used. </p>
  
  <p>A point I want to emphasize is that pooling takes multiple inputs and returns one based on some condition/function but that is vague, and it's usage in different scenarios can be for different goals (to an extent).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-21T03:02:03.410Z
Answers.Text: see that makes sense for images, but I saw a paper using it for words which didn't make sense to me
Answers.UserDisplayName: user8714896
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-21T03:05:35.107Z
Answers.Text: I gave the example across whole temporal dim but maybe imagine also using a finite region— which maintains tempora correlations and fits the idea like images
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-21T02:28:17.230Z
Answers.LastActivityDate: 2019-11-21T02:28:17.230Z
Answers.UserDisplayName: mshlis
Body: <p>I'm reading about max-pooling in a dynamic CNN paper. I can see how it can help find features in images, given that the pixel with the highest density gets pooled, but how does it help to find features in words?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-11T00:51:38.980Z
Comments.Text: Could you please provide the link to the paper you were reading?
Comments.UserDisplayName: nbro
Tags: <convolutional-neural-networks><pooling><max-pooling>
Title: How can max-pooling be applied to find features in words?
UserDisplayName: user8714896
ThreadText: # How can max-pooling be applied to find features in words?
  
  (Asked by: username_0 on 2019-11-21)
  
  I'm reading about max-pooling in a dynamic CNN paper. I can see how it can help find features in images, given that the pixel with the highest density gets pooled, but how does it help to find features in words?
  
  **Tags:** convolutional-neural-networks, pooling, max-pooling
  
  > **Comments:**
  > **username_1**: Could you please provide the link to the paper you were reading?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-11-21. Score: 1
  
  In an image you are pooling usually over some (n x n) set of positions which lets you maintain spatial correlation but on the other hand most 1D CNNs used for language modeling pool over the temporal axis completely annihilating all form of temporal correlation within the resulting feature vector. I will take it this difference is what confuses you.
  
  For simplicity sake Ill keep my explanation to max-pooling but it can extend to any of the others.
  
  MaxPooling as the name suggests just takes a maximum value from a set of candidates. In images you commonly see maxpooling over neighborhoods of pixels. This is because if we assume high activation is correlated to some feature being more prevalent (this assumption is not inherent, its more learned than anything else by construction of the architecture/training), then doing this we keep the most prevalent features, maintain spatial correlation, and reduce the dimensionality.
  
  When using 1D CNNs for LMs rather than RNNs the idea is that each kernel is *searching* for a feature, and so you use an exorbitant number of the kernels at various sizes to *search* for a lot of features. Pooling over the temporal dimension then just tells you at each neuron how prevalent its corresponding feature is in that time series input. That is why your output is usually a feature vector of size equal to the number of kernels you used.
  
  A point I want to emphasize is that pooling takes multiple inputs and returns one based on some condition/function but that is vague, and it's usage in different scenarios can be for different goals (to an extent).
  
  > **Comments:**
  > **username_0**: see that makes sense for images, but I saw a paper using it for words which didn't make sense to me  
  > **username_2**: I gave the example across whole temporal dim but maybe imagine also using a finite region— which maintains tempora correlations and fits the idea like images  
  
  ---
  
  