<p>I have two views of one object and using multi view cnn from .https://github.com/SAMY-ER/Multi-View-Image-Classification. This works great and I am getting better results. Though how can I apply grad cam to it. any help is appreciated . I want to see the cam on both views to see what parts in two images contributing to classification
This is my model</p>
<h1>MULTI-VIEW CONVOLUTIONAL NEURAL NETWORK (MVCNN) ARCHITECTURE</h1>
<blockquote>
<pre><code>class MVCNN(nn.Module):
   def __init__(self, num_classes=1000, pretrained=True):
    super(MVCNN, self).__init__()
   
    
    resnet = models.resnet50(pretrained = pretrained)
    
    fc_in_features = resnet.fc.in_features
    self.features = nn.Sequential(*list(resnet.children())[:-1])
    self.classifier = nn.Sequential(
        nn.Dropout(),
        ## multiplying by 2 to take care of two views
        nn.Linear(fc_in_features * 2, 2048),
        nn.ReLU(inplace=True),
        nn.Dropout(),
        nn.Linear(2048, 2048),
        nn.ReLU(inplace=True),
        nn.Linear(2048, num_classes)
    )
        
def forward(self, inputs): # inputs.shape = samples x views x height x width x channels
    inputs = inputs.transpose(0, 1)
    view_features = [] 
    for view_batch in inputs:
        view_batch = self.features(view_batch)
        view_batch = view_batch.view(view_batch.shape[0], view_batch.shape[1:].numel())
        view_features.append(view_batch)   
        
    concat_views = torch.cat(view_features,-1)
    #pooled_views, _ = torch.max(torch.stack(view_features), 0)
    #outputs = self.classifier(pooled_views)
    outputs = self.classifier(concat_views)
    return outputs
</code></pre>
</blockquote>

CC BY-SA 4.0
2023-05-08T11:57:43.220Z
grad-cam is calculated w.r.t. a target class - you should start with calculating the predictions score (usually `1-Predicted_score`) and backpropagating the error to a desired CNN layer
Aray Karjauv
CC BY-SA 4.0
2023-05-08T14:44:56.577Z
true, but i am merging two CNN layers from two different brances together. So, ideally i would like to see different activations in different branches/images. How do i do that?
user1631306
CC BY-SA 4.0
2023-05-08T16:06:45.823Z
I'm not familiar with this architecture but if I understand correctly you have different tails (output branches). In this case, you can backpropagate each branch separately to a shared CNN layer and then merge results or you could do the same for one of the last layers in each branch with the same shape (not sure in this case if it will make sense)
Aray Karjauv
CC BY-SA 4.0
2023-05-08T21:37:41.933Z
can you provide me an example/code?
user1631306
<convolutional-neural-networks><grad-cam>
Grad cam on multi view cnn
user1631306
# Grad cam on multi view cnn

(Asked by: username_0 on 2023-04-23)

I have two views of one object and using multi view cnn from .https://github.com/SAMY-ER/Multi-View-Image-Classification. This works great and I am getting better results. Though how can I apply grad cam to it. any help is appreciated . I want to see the cam on both views to see what parts in two images contributing to classification
This is my model
MULTI-VIEW CONVOLUTIONAL NEURAL NETWORK (MVCNN) ARCHITECTURE

class MVCNN(nn.Module):
   def __init__(self, num_classes=1000, pretrained=True):
    super(MVCNN, self).__init__()
   
    
    resnet = models.resnet50(pretrained = pretrained)
    
    fc_in_features = resnet.fc.in_features
    self.features = nn.Sequential(*list(resnet.children())[:-1])
    self.classifier = nn.Sequential(
        nn.Dropout(),
        ## multiplying by 2 to take care of two views
        nn.Linear(fc_in_features * 2, 2048),
        nn.ReLU(inplace=True),
        nn.Dropout(),
        nn.Linear(2048, 2048),
        nn.ReLU(inplace=True),
        nn.Linear(2048, num_classes)
    )
        
def forward(self, inputs): # inputs.shape = samples x views x height x width x channels
    inputs = inputs.transpose(0, 1)
    view_features = [] 
    for view_batch in inputs:
        view_batch = self.features(view_batch)
        view_batch = view_batch.view(view_batch.shape[0], view_batch.shape[1:].numel())
        view_features.append(view_batch)   
        
    concat_views = torch.cat(view_features,-1)
    #pooled_views, _ = torch.max(torch.stack(view_features), 0)
    #outputs = self.classifier(pooled_views)
    outputs = self.classifier(concat_views)
    return outputs

**Tags:** convolutional-neural-networks, grad-cam

> **Comments:**
> **username_1**: grad-cam is calculated w.r.t. a target class - you should start with calculating the predictions score (usually `1-Predicted_score`) and backpropagating the error to a desired CNN layer  
> **username_0**: true, but i am merging two CNN layers from two different brances together. So, ideally i would like to see different activations in different branches/images. How do i do that?  
> **username_1**: I'm not familiar with this architecture but if I understand correctly you have different tails (output branches). In this case, you can backpropagate each branch separately to a shared CNN layer and then merge results or you could do the same for one of the last layers in each branch with the same shape (not sure in this case if it will make sense)  
> **username_0**: can you provide me an example/code?  

---

## 0 answers

