Body: <p>I have two views of one object and using multi view cnn from .https://github.com/SAMY-ER/Multi-View-Image-Classification. This works great and I am getting better results. Though how can I apply grad cam to it. any help is appreciated . I want to see the cam on both views to see what parts in two images contributing to classification
  This is my model</p>
  <h1>MULTI-VIEW CONVOLUTIONAL NEURAL NETWORK (MVCNN) ARCHITECTURE</h1>
  <blockquote>
  <pre><code>class MVCNN(nn.Module):
     def __init__(self, num_classes=1000, pretrained=True):
      super(MVCNN, self).__init__()
     
      
      resnet = models.resnet50(pretrained = pretrained)
      
      fc_in_features = resnet.fc.in_features
      self.features = nn.Sequential(*list(resnet.children())[:-1])
      self.classifier = nn.Sequential(
          nn.Dropout(),
          ## multiplying by 2 to take care of two views
          nn.Linear(fc_in_features * 2, 2048),
          nn.ReLU(inplace=True),
          nn.Dropout(),
          nn.Linear(2048, 2048),
          nn.ReLU(inplace=True),
          nn.Linear(2048, num_classes)
      )
          
  def forward(self, inputs): # inputs.shape = samples x views x height x width x channels
      inputs = inputs.transpose(0, 1)
      view_features = [] 
      for view_batch in inputs:
          view_batch = self.features(view_batch)
          view_batch = view_batch.view(view_batch.shape[0], view_batch.shape[1:].numel())
          view_features.append(view_batch)   
          
      concat_views = torch.cat(view_features,-1)
      #pooled_views, _ = torch.max(torch.stack(view_features), 0)
      #outputs = self.classifier(pooled_views)
      outputs = self.classifier(concat_views)
      return outputs
  </code></pre>
  </blockquote>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-08T11:57:43.220Z
Comments.Text: grad-cam is calculated w.r.t. a target class - you should start with calculating the predictions score (usually `1-Predicted_score`) and backpropagating the error to a desired CNN layer
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-08T14:44:56.577Z
Comments.Text: true, but i am merging two CNN layers from two different brances together. So, ideally i would like to see different activations in different branches/images. How do i do that?
Comments.UserDisplayName: user1631306
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-08T16:06:45.823Z
Comments.Text: I'm not familiar with this architecture but if I understand correctly you have different tails (output branches). In this case, you can backpropagate each branch separately to a shared CNN layer and then merge results or you could do the same for one of the last layers in each branch with the same shape (not sure in this case if it will make sense)
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-08T21:37:41.933Z
Comments.Text: can you provide me an example/code?
Comments.UserDisplayName: user1631306
Tags: <convolutional-neural-networks><grad-cam>
Title: Grad cam on multi view cnn
UserDisplayName: user1631306
ThreadText: # Grad cam on multi view cnn
  
  (Asked by: username_0 on 2023-04-23)
  
  I have two views of one object and using multi view cnn from .https://github.com/SAMY-ER/Multi-View-Image-Classification. This works great and I am getting better results. Though how can I apply grad cam to it. any help is appreciated . I want to see the cam on both views to see what parts in two images contributing to classification
  This is my model
  
  # MULTI-VIEW CONVOLUTIONAL NEURAL NETWORK (MVCNN) ARCHITECTURE
  
  > ```
  > class MVCNN(nn.Module):
  >    def __init__(self, num_classes=1000, pretrained=True):
  >     super(MVCNN, self).__init__()
  >    
  >     
  >     resnet = models.resnet50(pretrained = pretrained)
  >     
  >     fc_in_features = resnet.fc.in_features
  >     self.features = nn.Sequential(*list(resnet.children())[:-1])
  >     self.classifier = nn.Sequential(
  >         nn.Dropout(),
  >         ## multiplying by 2 to take care of two views
  >         nn.Linear(fc_in_features * 2, 2048),
  >         nn.ReLU(inplace=True),
  >         nn.Dropout(),
  >         nn.Linear(2048, 2048),
  >         nn.ReLU(inplace=True),
  >         nn.Linear(2048, num_classes)
  >     )
  >         
  > def forward(self, inputs): # inputs.shape = samples x views x height x width x channels
  >     inputs = inputs.transpose(0, 1)
  >     view_features = [] 
  >     for view_batch in inputs:
  >         view_batch = self.features(view_batch)
  >         view_batch = view_batch.view(view_batch.shape[0], view_batch.shape[1:].numel())
  >         view_features.append(view_batch)   
  >         
  >     concat_views = torch.cat(view_features,-1)
  >     #pooled_views, _ = torch.max(torch.stack(view_features), 0)
  >     #outputs = self.classifier(pooled_views)
  >     outputs = self.classifier(concat_views)
  >     return outputs
  >
  > ```
  
  **Tags:** convolutional-neural-networks, grad-cam
  
  > **Comments:**
  > **username_1**: grad-cam is calculated w.r.t. a target class - you should start with calculating the predictions score (usually `1-Predicted_score`) and backpropagating the error to a desired CNN layer  
  > **username_0**: true, but i am merging two CNN layers from two different brances together. So, ideally i would like to see different activations in different branches/images. How do i do that?  
  > **username_1**: I'm not familiar with this architecture but if I understand correctly you have different tails (output branches). In this case, you can backpropagate each branch separately to a shared CNN layer and then merge results or you could do the same for one of the last layers in each branch with the same shape (not sure in this case if it will make sense)  
  > **username_0**: can you provide me an example/code?  
  
  ---
  
  ## 0 answers
  
  