Answers.Body: <h3>TL;DR:</h3>
  <p>It's definitely worth trying to benefit from the learned features from the ResNet. As it's made of mainly <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">pretrained convolutional layers with pooling</a>, adding new resizing layers upfront is unnecessary and would very likely hurt the performance. On the plus side, this means you should be able to approach the problem with your original resolution and fine-tune the exisiting ResNet with you own custom output layers.</p>
  <h3>More Detail:</h3>
  <p>Off the bat, Convolutional layers tend to learn to work with different scales of features through the model, with early layers working with pixel patterns and later layers finding more abstract patterns <a href="https://distill.pub/2017/feature-visualization/" rel="nofollow noreferrer">as discussed in this wonderful Distill piece</a>, which contains the image below:</p>
  <p><a href="https://i.sstatic.net/1JzAQ.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1JzAQ.jpg" alt="Screenshot of the multiple-layer visualisation the GoogLeNet model's feature activations. Progressing from left to right, the model visualises edgse, textures, patterns, parts, and finally objects and concepts. Sourced from: https://distill.pub/2017/feature-visualization/" /></a></p>
  <p>Adding convolutional layers before the input of a trained <em>backbone</em> model will add a layer of abstraction from the original pixels which the early layers of the <em>backbone</em> won't be trained to work on. As such, the additional layers you suggest in the question would likely <strong>interfere quite heavily with the backbone's performance</strong>.</p>
  <hr />
  <p>However, as the majority of the ResNet18 architecture is convolutional layers it's possible (as you may already be aware) to feed any size image into the model and get a prediction on the output. This is due to the convolutional and global pooling layers <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="nofollow noreferrer">reshaping the input as it goes through the model</a>, discussed briefly <a href="https://stackoverflow.com/q/73775996/20143675">here</a>, which means you won't lose pixel-resolution at the input stage.</p>
  <p>This means you could follow the traditional fine-tuning route using the ResNet18 archicture as-is, <a href="https://discuss.pytorch.org/t/how-to-modify-the-final-fc-layer-based-on-the-torch-model/766" rel="nofollow noreferrer"><strong>replacing and retraining</strong> the final fully-connected layer</a>, <strong>freezing</strong> the rest of the parameters to speed up the training.</p>
  <p>That said, due to the difference between the original and your image resolutions it's possible that the performance of the <em>backbones</em>'s convolutions will be degraded regardless. To mitigate this, you could <strong>not freeze any of the ResNet18 parameters</strong>, effectively using it as a <a href="https://stats.stackexchange.com/q/193082">pre-trained</a> warm-start for you problem and leveraging <a href="https://en.wikipedia.org/wiki/Transfer_learning" rel="nofollow noreferrer">transfer learning</a>.</p>
  <p>Happy experimenting!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-03T12:39:12.167Z
Answers.LastActivityDate: 2022-10-03T12:39:12.167Z
Answers.UserDisplayName: James Ashford
Body: <p>I have a ResNet18 model trained on the Places365 image dataset, and I'd like to use this pre-trained model to expedite the training needed to identify distressed houses. My dataset is images of size 800x800x3, but the inputs are of size 224x224x3. I do not want to resize the image to 224x224 since I'm worried it will lose house distress indicators (chipped paint and loose shingles) during conversion.</p>
  <p>My idea was to add extra layers that can handle the larger images before feeding them into ResNet. I have the following pytorch model:</p>
  <pre><code>import torch
  from torch import nn
  from torchvision import models
  
  class NewModel(nn.Model):
      def __init__(self, pretrain_model_path) -&gt; None:
          # Not sure here
          self.pre_layers = None
          # import the trained model
          model = models.resnet18(num_classes=365)
          checkpoint = torch.load(pretrain_model_path, map_location=lambda storage, loc: storage)
          state_dict = {str.replace(k,'module.',''): v for k,v in checkpoint['state_dict'].items()}
          model.load_state_dict(state_dict)
          # change prediction class count
          model.fc = nn.Linear(model.fc.in_features, 4)
          self.backbone = model
  
      def forward(self, x):
          x = self.pre_layers(x)
          x = self.backbone(x)
          return x
  </code></pre>
  <p>Is this a common practice or is it better to make one from scratch that is built for this size image and problem specifically? How would I go about implementing it if so?</p>
  
Tags: <convolutional-neural-networks><pytorch><residual-networks>
Title: How can I use larger input images when using a pre-trained CNN without resizing?
UserDisplayName: Cole Tritch
ThreadText: # How can I use larger input images when using a pre-trained CNN without resizing?
  
  (Asked by: username_0 on 2022-10-03)
  
  I have a ResNet18 model trained on the Places365 image dataset, and I'd like to use this pre-trained model to expedite the training needed to identify distressed houses. My dataset is images of size 800x800x3, but the inputs are of size 224x224x3. I do not want to resize the image to 224x224 since I'm worried it will lose house distress indicators (chipped paint and loose shingles) during conversion.
  My idea was to add extra layers that can handle the larger images before feeding them into ResNet. I have the following pytorch model:
  import torch
  from torch import nn
  from torchvision import models
  
  class NewModel(nn.Model):
      def __init__(self, pretrain_model_path) -> None:
          # Not sure here
          self.pre_layers = None
          # import the trained model
          model = models.resnet18(num_classes=365)
          checkpoint = torch.load(pretrain_model_path, map_location=lambda storage, loc: storage)
          state_dict = {str.replace(k,'module.',''): v for k,v in checkpoint['state_dict'].items()}
          model.load_state_dict(state_dict)
          # change prediction class count
          model.fc = nn.Linear(model.fc.in_features, 4)
          self.backbone = model
  
      def forward(self, x):
          x = self.pre_layers(x)
          x = self.backbone(x)
          return x
  
  Is this a common practice or is it better to make one from scratch that is built for this size image and problem specifically? How would I go about implementing it if so?
  
  **Tags:** convolutional-neural-networks, pytorch, residual-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-03. Score: 3 | Accepted answer
  
  ### TL;DR:
  
  It's definitely worth trying to benefit from the learned features from the ResNet. As it's made of mainly [pretrained convolutional layers with pooling](https://arxiv.org/pdf/1512.03385.pdf), adding new resizing layers upfront is unnecessary and would very likely hurt the performance. On the plus side, this means you should be able to approach the problem with your original resolution and fine-tune the exisiting ResNet with you own custom output layers.
  
  ### More Detail:
  
  Off the bat, Convolutional layers tend to learn to work with different scales of features through the model, with early layers working with pixel patterns and later layers finding more abstract patterns [as discussed in this wonderful Distill piece](https://distill.pub/2017/feature-visualization/), which contains the image below:
  
  [![Screenshot of the multiple-layer visualisation the GoogLeNet model's feature activations. Progressing from left to right, the model visualises edgse, textures, patterns, parts, and finally objects and concepts. Sourced from: https://distill.pub/2017/feature-visualization/](https://i.sstatic.net/1JzAQ.jpg)](https://i.sstatic.net/1JzAQ.jpg)
  
  Adding convolutional layers before the input of a trained *backbone* model will add a layer of abstraction from the original pixels which the early layers of the *backbone* won't be trained to work on. As such, the additional layers you suggest in the question would likely **interfere quite heavily with the backbone's performance**.
  
  ---
  
  However, as the majority of the ResNet18 architecture is convolutional layers it's possible (as you may already be aware) to feed any size image into the model and get a prediction on the output. This is due to the convolutional and global pooling layers [reshaping the input as it goes through the model](https://arxiv.org/pdf/1512.03385.pdf), discussed briefly [here](https://stackoverflow.com/q/73775996/20143675), which means you won't lose pixel-resolution at the input stage.
  
  This means you could follow the traditional fine-tuning route using the ResNet18 archicture as-is, [**replacing and retraining** the final fully-connected layer](https://discuss.pytorch.org/t/how-to-modify-the-final-fc-layer-based-on-the-torch-model/766), **freezing** the rest of the parameters to speed up the training.
  
  That said, due to the difference between the original and your image resolutions it's possible that the performance of the *backbones*'s convolutions will be degraded regardless. To mitigate this, you could **not freeze any of the ResNet18 parameters**, effectively using it as a [pre-trained](https://stats.stackexchange.com/q/193082) warm-start for you problem and leveraging [transfer learning](https://en.wikipedia.org/wiki/Transfer_learning).
  
  Happy experimenting!
  
  
  ---
  
  