Body: <p>I have a low resolution thermal/IR image (for example 32x32 or 80x64) and a high resolution webcam image. I would like to combine the two to &quot;fake&quot; a high resolution thermal image (I can already map them together via homography). One could probably just apply a FLIR-like palette to the IR image, scale it up, and combine it with the brightness channel of the visible spectrum image. But that would of cause visible artifacts at the pixel edges of the IR picture.</p>
  <p>I wonder if there is an AI based approach to colorize the webcam image with the IR data. When a warm IR pixel partially covers a person and partially the background, it would only color the &quot;person&quot; warm, and take the &quot;background&quot; color from the neighboring IR pixel. For this it would have to consider a small vicinity of either picture at a time.</p>
  <p>Although I'm familiar with machine learning in the context of multivariate analysis and classification, I have no experience with modern deep learning or AI based image processing. I would guess that something like style transfer would be a starting point for what I'm trying to achive. One would need 1) a way to identify features (like foreground/background, person/wall) and 2) a way to combine these features with the IR truth to result in a colorized bitmap, I assume.</p>
  <p>What would be the best approach to do this? Maybe this is already a solved problem - I have a feeling this might already be a solved problem. In any case I would be grateful for literature pointers.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-11T20:50:59.383Z
Comments.Text: This sounds like a segmentation task. Take a look at U-Net. If you have one-to-one IR and webcam images, you can then upscale the resulting map to the webcam image. This also could be done with the Pix2Pix network (which is also based on U-Net)
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-12T17:11:36.503Z
Comments.Text: Thanks, U-Net and Pix2Pix look pretty promising. The only issue is that they are focused on segmentation, whereas I don't think I always have clear boundaries. In my case there can also be "gradients" (think a wall diagonally to the camera) or soft bodies. But it is probably a good starting point.
Comments.UserDisplayName: jdm
Tags: <computer-vision><style-transfer>
Title: Upscaling a low-res IR image with a high res webcam image
UserDisplayName: jdm
ThreadText: # Upscaling a low-res IR image with a high res webcam image
  
  (Asked by: username_0 on 2021-07-11)
  
  I have a low resolution thermal/IR image (for example 32x32 or 80x64) and a high resolution webcam image. I would like to combine the two to "fake" a high resolution thermal image (I can already map them together via homography). One could probably just apply a FLIR-like palette to the IR image, scale it up, and combine it with the brightness channel of the visible spectrum image. But that would of cause visible artifacts at the pixel edges of the IR picture.
  
  I wonder if there is an AI based approach to colorize the webcam image with the IR data. When a warm IR pixel partially covers a person and partially the background, it would only color the "person" warm, and take the "background" color from the neighboring IR pixel. For this it would have to consider a small vicinity of either picture at a time.
  
  Although I'm familiar with machine learning in the context of multivariate analysis and classification, I have no experience with modern deep learning or AI based image processing. I would guess that something like style transfer would be a starting point for what I'm trying to achive. One would need 1) a way to identify features (like foreground/background, person/wall) and 2) a way to combine these features with the IR truth to result in a colorized bitmap, I assume.
  
  What would be the best approach to do this? Maybe this is already a solved problem - I have a feeling this might already be a solved problem. In any case I would be grateful for literature pointers.
  
  **Tags:** computer-vision, style-transfer
  > **Comments:**
  > **username_1**: This sounds like a segmentation task. Take a look at U-Net. If you have one-to-one IR and webcam images, you can then upscale the resulting map to the webcam image. This also could be done with the Pix2Pix network (which is also based on U-Net)  
  > **username_0**: Thanks, U-Net and Pix2Pix look pretty promising. The only issue is that they are focused on segmentation, whereas I don't think I always have clear boundaries. In my case there can also be "gradients" (think a wall diagonally to the camera) or soft bodies. But it is probably a good starting point.  
  
  ---
  
  ## 0 answers
  
  