Answers.Body: <p>You would expect the model to extrapolate due to the 1D-Convolutions and the pooling, because they are translation equivariant (with local invariance due to the pooling). Thus, these layers produce encodings that are shifted just like the input timeseries. In your implementation, these operations are currently doing the following: The convolutions + pooling layers have a receptive field of <span class="math-container">$144$</span> frequencies</p>
  <p><span class="math-container">$$
  \text{Convolution}: \;\;\text{kernel-size} = 128 ; \; \; \text{stride} = 1\\
  \text{Max-Pooling}: \;\;\text{kernel-size} = 16 ; \; \; \text{stride} = 2\\
  \Rightarrow \text{Receptive Field} \approx 2 \cdot \frac{128}{2} + 16 = 144
  $$</span></p>
  <p>and conceptually, this means that you encode segments with 144 <em>steps</em> into a single step. You then feed the resulting sequence of 4026 steps (<span class="math-container">$257664 \;/\; 64 = 4026$</span>) to a dense layer. This dense layer is translation sensitive and therefore doesn't naturally generalize to shifted timeseries.</p>
  <p>One possible way to approach this is to make the model more translation equivariant. For example by stacking more convolutions/pooling layers to increase the receptive field of your convolutional encoder part. As a side effect, this will probably also benefit training, as the dense layer will get a much smaller input vector.</p>
  <p>Another/Additional option is to replace the <code>flatten</code> operator (translation sensitive) with a global pooling operation (translation invariant). But this option will only work if the receptive field of your encoder is wide enough to capture the pattern that identified either class.</p>
  <p><strong>EDIT</strong> Overall I would look to change the setup of the convolutional encoder: The 1D filters of the convolution have a large overlap (due to the stride of 1 and the kernel size of 128). This results in unnecessarily large feature maps, so you can probably increase the strides a lot (even as large as 32-96 should not be a problem). Also to make the model generalize better to the patterns in your data, I'd try stacking more than one convolution.</p>
  <p>I hope this was helpful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T01:12:48.990Z
Answers.Text: The model performs worse on the training data as I add more convolutional and pooling layers. Would this be because the kernal sizes and output channels are not selected properly?  Should the training data be examples that are largely centered (i.e., same frequency, varying levels of amplitude), or does it need to include all the various offsets?
Answers.UserDisplayName: BigBrownBear00
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T09:42:53.727Z
Answers.Text: This can have many reasons, without knowing your dataset it's hard to be more specific about hyperparameters. Also, these things are largely intuition and ultimately only experiments will show what works best. Whether you should center your samples or not depends on how you want to use your model. If it's fine for the use case to center the data, then that can help.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T09:42:56.230Z
Answers.Text: It also depends on the discriminative feature in a data sample.   For example: If the classes are determined by shape, then centering should be optional (if the model is setup correctly). If the classes are determined by position, then centering could prevent the model from learning anything,
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T09:45:22.380Z
Answers.Text: Things I notice about your model is that your convolutions use a large kernel size and a stride of 1. The stride can be set to a higher value if you use a kernel size this large (even in the range of 32-96 in your case). This is because a large part of the kernel will overlap and wont help a lot. If you increase your strides, more convolutional layers will probably also help you. But you might want to specify your data a bit more in detail to get better help.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T13:00:59.917Z
Answers.Text: The classes are determined by shape, not position. Training models on shape when centered works exceptionally well. It's when a random offset is introduced that things fall apart. 
  
  Increasing the stride seems to help a lot. I will need to run more tests and experiment with the hyperparameters but this is encouraging.
Answers.UserDisplayName: BigBrownBear00
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T13:01:16.217Z
Answers.Text: Happy to provide any more information you need about the data. What would you like to know?
Answers.UserDisplayName: BigBrownBear00
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T13:20:55.370Z
Answers.Text: I'm glad this helped. 
  I would look at different data samples and intuitively decide on my architecture. Here are some rules of thumb that I usually follow: First, what's important to look at is the granularity of detail of the patterns that the model can encounter. If you have a high level of detail (e.g. in real world photographs or voice recordings), then a small kernel-size usually works better. If your details are rather coarse (e.g. images of schematics, or the example that you provide), you can get away with larger kernel-size (and stride).
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T13:22:19.173Z
Answers.Text: Second, try to look at composition of the patterns: If your patterns are assembled from only a few reoccurring primitives (e.g. images of schematics, or the example that you provide), then a shallow encoder will usually be sufficient. If you patterns are composed from a variety of very different primitives (e.g. in real world photographs or voice recordings), then I would go deeper.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T13:22:21.250Z
Answers.Text: Third, if translation of the shape along the frequency axis does not matter, try to replace the flatten with a translation invariant operations like global pooling. To get this to work you should increase the receptive field of your convolutional encoder a bit mode so that the discriminative shapes are covered
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-12T04:07:04.833Z
Answers.Text: When I add the global pooling operation using `nn.AdaptiveAvgPool1d(1)` in PyTorch I get dimension issues going into the final Linear layer. Don't I need to flatten before going into the Linear layer?
Answers.UserDisplayName: BigBrownBear00
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-13T09:35:20.093Z
Answers.Text: It should work: The Flatten operation takes an input vector of arbitrary shape and unpacks the innermost dimension. You have the input shape `[batch, time, 64]` and after flatten yuo should get `[batch, time * 64]`. What you want from the pooling is that it takes either the max or the mean over the second axis. With the current model setup this should yield  `[batch, 64]`. Try to set the input dim for the dense layer to 64 and it should be fine.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-13T09:40:55.960Z
Answers.Text: To do it in PyTorch, you can use the [`Tensor.max()`](https://pytorch.org/docs/stable/generated/torch.max.html#torch.max) operation on the second axis.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-13T13:05:24.590Z
Answers.Text: I want to accept your answer but could you first update it to include some of the details from our discussion in the comments? The key pieces that helped make the model translation invariant were: 1) larger strides for the large kernal sizes. 2) more conv layers (use 3-4). 3) train with data that includes off centered offsets.
Answers.UserDisplayName: BigBrownBear00
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-13T13:05:36.393Z
Answers.Text: Regarding #3, when training with "centered" data, I was able to get decent performance on non-centered test data, but the best performance was when I trained with examples that included off centered data. 
  
  I tried messing with AdapativeAvgPool, not sure it helps a ton, so it's unclear how important that piece is.
Answers.UserDisplayName: BigBrownBear00
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T11:19:16.770Z
Answers.Text: So the reasoning to use the global pooling instead of the flatten is to completely ignore the frequency band of the patterns. Doing this should yield a model with out-of-the-box shift generalization if everything else is setup nicely. For this to work, the convolutional encoder part does all the work and has to be able to capture discriminative patterns. I would try to get it to work, but that's of course up to you and depends on the dataset.
  Aand I will update my answer.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T12:41:37.627Z
Answers.Text: I want to get global pooling to work. I will make another question because I am seeing conflicting information on how to implement it. Whether there should be a flatten/linear after the global pooling layer, or not, and how to properly use it.
Answers.UserDisplayName: BigBrownBear00
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T12:49:46.927Z
Answers.Text: I use TensorFlow usually, but if I understand [the torch docs](https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool1d.html) correctly, the Adaptive Pooling you use doesn't collapse the pooling dimension (there will be an axis of size 1, for example `[batch, time, dim] -> [batch, 1, dim]`), so yes, you need a flatten layer afterwards to remove this singular dimension.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T12:56:48.967Z
Answers.Text: Adding a Flatten at the end at least gets my model running. And I omit the final Linear layer, correct?
Answers.UserDisplayName: BigBrownBear00
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T13:05:02.937Z
Answers.Text: I would keep it, but I think for this decision only experiments will show what works best.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T12:47:58.593Z
Answers.Text: Update: I experimented training the model using only centered data with different depths (i.e., number of conv layers) and was able to get good performance even with one layer as long as I used global pooling. Without global pooling the model stopped being translation invariant. Thank you again so much for the help!
Answers.UserDisplayName: BigBrownBear00
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T15:44:14.210Z
Answers.Text: Very cool - thanks for the update! I'm glad I could help and also that you got it running nicely :)
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T22:39:23.450Z
Answers.LastActivityDate: 2023-01-15T11:24:51.400Z
Answers.LastEditDate: 2023-01-15T11:24:51.400Z
Answers.UserDisplayName: Chillston
Body: <p>I have trained a CNN on one dimensional data that is the power spectral density (PSD) of a <span class="math-container">$N$</span> different classes of signals (<span class="math-container">$N=4$</span>). Each of the <span class="math-container">$N$</span> signals has a different spectral shape (not shown here). For illustrative purposes, the plots shown below are from the same signal class. The idea is to treat this as an image classification problem.</p>
  <p>The model performs exceptionally well when the training data has all the examples centered around the same frequency (i.e., within a few hundred Hertz): <a href="https://i.sstatic.net/VDQdz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VDQdz.png" alt="All training examples centered on or around 0 Hz" /></a></p>
  <p>The CNN fails to properly classify new examples that are outside the frequency range of the training data:</p>
  <p><a href="https://i.sstatic.net/1TfW3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1TfW3.png" alt="Invalid classification, new data outside frequency range of trained data" /></a></p>
  <p><strong>Model Details and Assumptions:</strong></p>
  <p>The CNN model is implemented in PyTorch using the following layers:</p>
  <pre><code>    model = nn.Sequential(
              nn.Conv1d(in_channels=1, out_channels=64, kernel_size=128, stride=1, padding=1),
              nn.ReLU(),
              nn.MaxPool1d(kernel_size=16, stride=2),
              nn.Dropout(p=0.25),
              nn.Flatten(),
              nn.Linear(257664, n_classes)
      )
  </code></pre>
  <p>The optimizer is <code>torch.optim.Adam</code>. Batch size has been varied from 8 to 128, epochs varied from 10 to 50. The input data is normalized to <span class="math-container">$[0,1]$</span>. Training examples varied from [2000, 8000], where 20% are used for validation and 20% are used for test.
  I have also tried adding additional convolutional layers, varying the kernal sizes, neurons, layers, etc.</p>
  <p><strong>Questions:</strong></p>
  <ol>
  <li>Shouldn't the CNN model generalize such that new examples that are not within the same frequency range (i.e., centered around the middle) should be identified as the correct class?</li>
  <li>Are there other steps I need to take, whether in the model or training data?</li>
  </ol>
  
Tags: <neural-networks><machine-learning><convolutional-neural-networks><image-recognition>
Title: Should a CNN generalize to arbitrary positions in the data?
UserDisplayName: BigBrownBear00
ThreadText: # Should a CNN generalize to arbitrary positions in the data?
  
  (Asked by: username_0 on 2023-01-07)
  
  I have trained a CNN on one dimensional data that is the power spectral density (PSD) of a $N$ different classes of signals ($N=4$). Each of the $N$ signals has a different spectral shape (not shown here). For illustrative purposes, the plots shown below are from the same signal class. The idea is to treat this as an image classification problem.
  The model performs exceptionally well when the training data has all the examples centered around the same frequency (i.e., within a few hundred Hertz): 
  The CNN fails to properly classify new examples that are outside the frequency range of the training data:
  
  Model Details and Assumptions:
  The CNN model is implemented in PyTorch using the following layers:
      model = nn.Sequential(
              nn.Conv1d(in_channels=1, out_channels=64, kernel_size=128, stride=1, padding=1),
              nn.ReLU(),
              nn.MaxPool1d(kernel_size=16, stride=2),
              nn.Dropout(p=0.25),
              nn.Flatten(),
              nn.Linear(257664, n_classes)
      )
  
  The optimizer is torch.optim.Adam. Batch size has been varied from 8 to 128, epochs varied from 10 to 50. The input data is normalized to $[0,1]$. Training examples varied from [2000, 8000], where 20% are used for validation and 20% are used for test.
  I have also tried adding additional convolutional layers, varying the kernal sizes, neurons, layers, etc.
  Questions:
  
  Shouldn't the CNN model generalize such that new examples that are not within the same frequency range (i.e., centered around the middle) should be identified as the correct class?
  Are there other steps I need to take, whether in the model or training data?
  
  **Tags:** neural-networks, machine-learning, convolutional-neural-networks, image-recognition
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-09. Score: 2 | Accepted answer
  
  You would expect the model to extrapolate due to the 1D-Convolutions and the pooling, because they are translation equivariant (with local invariance due to the pooling). Thus, these layers produce encodings that are shifted just like the input timeseries. In your implementation, these operations are currently doing the following: The convolutions + pooling layers have a receptive field of $144$ frequencies
  
  $$
  \text{Convolution}: \;\;\text{kernel-size} = 128 ; \; \; \text{stride} = 1\\
  \text{Max-Pooling}: \;\;\text{kernel-size} = 16 ; \; \; \text{stride} = 2\\
  \Rightarrow \text{Receptive Field} \approx 2 \cdot \frac{128}{2} + 16 = 144
  $$
  
  and conceptually, this means that you encode segments with 144 *steps* into a single step. You then feed the resulting sequence of 4026 steps ($257664 \;/\; 64 = 4026$) to a dense layer. This dense layer is translation sensitive and therefore doesn't naturally generalize to shifted timeseries.
  
  One possible way to approach this is to make the model more translation equivariant. For example by stacking more convolutions/pooling layers to increase the receptive field of your convolutional encoder part. As a side effect, this will probably also benefit training, as the dense layer will get a much smaller input vector.
  
  Another/Additional option is to replace the `flatten` operator (translation sensitive) with a global pooling operation (translation invariant). But this option will only work if the receptive field of your encoder is wide enough to capture the pattern that identified either class.
  
  **EDIT** Overall I would look to change the setup of the convolutional encoder: The 1D filters of the convolution have a large overlap (due to the stride of 1 and the kernel size of 128). This results in unnecessarily large feature maps, so you can probably increase the strides a lot (even as large as 32-96 should not be a problem). Also to make the model generalize better to the patterns in your data, I'd try stacking more than one convolution.
  
  I hope this was helpful.
  
  > **Comments:**
  > **username_0**: The model performs worse on the training data as I add more convolutional and pooling layers. Would this be because the kernal sizes and output channels are not selected properly?  Should the training data be examples that are largely centered (i.e., same frequency, varying levels of amplitude), or does it need to include all the various offsets?  
  > **username_1**: This can have many reasons, without knowing your dataset it's hard to be more specific about hyperparameters. Also, these things are largely intuition and ultimately only experiments will show what works best. Whether you should center your samples or not depends on how you want to use your model. If it's fine for the use case to center the data, then that can help.  
  > **username_1**: It also depends on the discriminative feature in a data sample.   For example: If the classes are determined by shape, then centering should be optional (if the model is setup correctly). If the classes are determined by position, then centering could prevent the model from learning anything,  
  > **username_1**: Things I notice about your model is that your convolutions use a large kernel size and a stride of 1. The stride can be set to a higher value if you use a kernel size this large (even in the range of 32-96 in your case). This is because a large part of the kernel will overlap and wont help a lot. If you increase your strides, more convolutional layers will probably also help you. But you might want to specify your data a bit more in detail to get better help.  
  > **username_0**: The classes are determined by shape, not position. Training models on shape when centered works exceptionally well. It's when a random offset is introduced that things fall apart.   Increasing the stride seems to help a lot. I will need to run more tests and experiment with the hyperparameters but this is encouraging.  
  > **username_0**: Happy to provide any more information you need about the data. What would you like to know?  
  > **username_1**: I'm glad this helped.  I would look at different data samples and intuitively decide on my architecture. Here are some rules of thumb that I usually follow: First, what's important to look at is the granularity of detail of the patterns that the model can encounter. If you have a high level of detail (e.g. in real world photographs or voice recordings), then a small kernel-size usually works better. If your details are rather coarse (e.g. images of schematics, or the example that you provide), you can get away with larger kernel-size (and stride).  
  > **username_1**: Second, try to look at composition of the patterns: If your patterns are assembled from only a few reoccurring primitives (e.g. images of schematics, or the example that you provide), then a shallow encoder will usually be sufficient. If you patterns are composed from a variety of very different primitives (e.g. in real world photographs or voice recordings), then I would go deeper.  
  > **username_1**: Third, if translation of the shape along the frequency axis does not matter, try to replace the flatten with a translation invariant operations like global pooling. To get this to work you should increase the receptive field of your convolutional encoder a bit mode so that the discriminative shapes are covered  
  > **username_0**: When I add the global pooling operation using `nn.AdaptiveAvgPool1d(1)` in PyTorch I get dimension issues going into the final Linear layer. Don't I need to flatten before going into the Linear layer?  
  > **username_1**: It should work: The Flatten operation takes an input vector of arbitrary shape and unpacks the innermost dimension. You have the input shape `[batch, time, 64]` and after flatten yuo should get `[batch, time * 64]`. What you want from the pooling is that it takes either the max or the mean over the second axis. With the current model setup this should yield  `[batch, 64]`. Try to set the input dim for the dense layer to 64 and it should be fine.  
  > **username_1**: To do it in PyTorch, you can use the [`Tensor.max()`](https://pytorch.org/docs/stable/generated/torch.max.html#torch.max) operation on the second axis.  
  > **username_0**: I want to accept your answer but could you first update it to include some of the details from our discussion in the comments? The key pieces that helped make the model translation invariant were: 1) larger strides for the large kernal sizes. 2) more conv layers (use 3-4). 3) train with data that includes off centered offsets.  
  > **username_0**: Regarding #3, when training with "centered" data, I was able to get decent performance on non-centered test data, but the best performance was when I trained with examples that included off centered data.   I tried messing with AdapativeAvgPool, not sure it helps a ton, so it's unclear how important that piece is.  
  > **username_1**: So the reasoning to use the global pooling instead of the flatten is to completely ignore the frequency band of the patterns. Doing this should yield a model with out-of-the-box shift generalization if everything else is setup nicely. For this to work, the convolutional encoder part does all the work and has to be able to capture discriminative patterns. I would try to get it to work, but that's of course up to you and depends on the dataset. Aand I will update my answer.  
  > **username_0**: I want to get global pooling to work. I will make another question because I am seeing conflicting information on how to implement it. Whether there should be a flatten/linear after the global pooling layer, or not, and how to properly use it.  
  > **username_1**: I use TensorFlow usually, but if I understand [the torch docs](https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool1d.html) correctly, the Adaptive Pooling you use doesn't collapse the pooling dimension (there will be an axis of size 1, for example `[batch, time, dim] -> [batch, 1, dim]`), so yes, you need a flatten layer afterwards to remove this singular dimension.  
  > **username_0**: Adding a Flatten at the end at least gets my model running. And I omit the final Linear layer, correct?  
  > **username_1**: I would keep it, but I think for this decision only experiments will show what works best.  
  > **username_0**: Update: I experimented training the model using only centered data with different depths (i.e., number of conv layers) and was able to get good performance even with one layer as long as I used global pooling. Without global pooling the model stopped being translation invariant. Thank you again so much for the help!  
  > **username_1**: Very cool - thanks for the update! I'm glad I could help and also that you got it running nicely :)  
  
  ---
  
  