Answers.Body: <p>Padding is indeed the easiest solution. And if no bias is used then masking the extra values during the loss computation is also not necessary, since it's enough to use zero as padding value.</p>
  <p>You might be interested though in checking <a href="https://arxiv.org/pdf/1406.4729.pdf" rel="nofollow noreferrer">Spatial Pyramid Pooling</a>. This pooling method allows to combine fully convolutional modules and dense layers, i.e, it can be initialized to produce a specific fixed output size while allowing varying input sizes, for both, training and inference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-06T01:47:52.320Z
Answers.Text: Thanks for the answer and for the Spatial Pyramid Pooling recommendation!
Answers.UserDisplayName: NateW
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-06T00:42:30.800Z
Answers.LastActivityDate: 2021-11-06T15:23:09.810Z
Answers.LastEditDate: 2021-11-06T15:23:09.810Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I'm working on a depth estimation network.  It has two outputs:</p>
  <ol>
  <li>A relative depth map</li>
  <li>A scalar for scaling the relative depth map into an absolute depth map.  This second output uses dense layers so we cannot use variable-sized input.</li>
  </ol>
  <p>We are trying to handle two different dimensions (192x256 and 256x192). The current approach is to letterbox the image, meaning apply black on the image so that it comes out to 256x256. We decided on this approach instead of center-cropping images to 192x192 because we believe we may lose valuable data with cropping.</p>
  <p>When using letterboxes, I see two paths:</p>
  <ol>
  <li>Ignore the letterbox portions of the image in my loss function.  The loss function will only perform calculations on the original portion of the image.</li>
  <li>Set a static value for the letterbox portion and include it as part of the loss.</li>
  </ol>
  <p>Is #1 the correct approach?  The network will then be able to predict any depth value for the black letterbox portions without being penalized. I'm concerned with #2 about confusing the network between the letterbox portion and actual dark portions of images.</p>
  
Tags: <deep-learning><convolutional-neural-networks><data-preprocessing><data-labelling><fully-convolutional-networks>
Title: Best practice for handling letterboxed images for non fully-convolutional deep learning networks?
UserDisplayName: NateW
ThreadText: # Best practice for handling letterboxed images for non fully-convolutional deep learning networks?
  
  (Asked by: username_0 on 2021-11-05)
  
  I'm working on a depth estimation network. It has two outputs:
  
  1. A relative depth map
  2. A scalar for scaling the relative depth map into an absolute depth map. This second output uses dense layers so we cannot use variable-sized input.
  
  We are trying to handle two different dimensions (192x256 and 256x192). The current approach is to letterbox the image, meaning apply black on the image so that it comes out to 256x256. We decided on this approach instead of center-cropping images to 192x192 because we believe we may lose valuable data with cropping.
  
  When using letterboxes, I see two paths:
  
  1. Ignore the letterbox portions of the image in my loss function. The loss function will only perform calculations on the original portion of the image.
  2. Set a static value for the letterbox portion and include it as part of the loss.
  
  Is #1 the correct approach? The network will then be able to predict any depth value for the black letterbox portions without being penalized. I'm concerned with #2 about confusing the network between the letterbox portion and actual dark portions of images.
  
  **Tags:** deep-learning, convolutional-neural-networks, data-preprocessing, data-labelling, fully-convolutional-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-06. Score: 2 | Accepted answer
  
  Padding is indeed the easiest solution. And if no bias is used then masking the extra values during the loss computation is also not necessary, since it's enough to use zero as padding value.
  
  You might be interested though in checking [Spatial Pyramid Pooling](https://arxiv.org/pdf/1406.4729.pdf). This pooling method allows to combine fully convolutional modules and dense layers, i.e, it can be initialized to produce a specific fixed output size while allowing varying input sizes, for both, training and inference.
  
  > **Comments:**
  > **username_0**: Thanks for the answer and for the Spatial Pyramid Pooling recommendation!  
  
  ---
  
  