Hello,
In this pull request you can see the work-around for t5.

  
      github.com/huggingface/transformers
  
  
    
  
    
  

  
    
      Return cross-attention from T5 models
    

    
      huggingface:master ← noahtren:master
    

    
      
        opened 
        
          
        
        Sep 18, 2020
      
      

      
        
          
          noahtren
        
      

      
        
          +19
          -28
        
      
    

  


  
  
    
    
  
  


So in your case, I guess if you make the following changes:

  
      github.com
  
  
    huggingface/transformers/blob/master/src/transformers/modeling_bart.py#L460 11

x = F.dropout(x, p=self.dropout, training=self.training)
x = residual + x
if not self.normalize_before:
    x = self.self_attn_layer_norm(x)

# Cross attention
residual = x
assert self.encoder_attn.cache_key != self.self_attn.cache_key
if self.normalize_before:
    x = self.encoder_attn_layer_norm(x)
x, _ = self.encoder_attn(
    query=x,
    key=encoder_hidden_states,
    key_padding_mask=encoder_attn_mask,
    layer_state=layer_state,  # mutates layer state
)
x = F.dropout(x, p=self.dropout, training=self.training)
x = residual + x
if not self.normalize_before:
    x = self.encoder_attn_layer_norm(x)




  
  
    
    
  
  



add the _ variable (which holds cross-attentions) to the returned attention value, you can get layer-wise cross attentions scores when output_attentions=True.


  
      github.com
  
  
    huggingface/transformers/blob/master/src/transformers/modeling_bart.py#L484 5

            x = self.final_layer_norm(x)
        x = self.activation_fn(self.fc1(x))
        x = F.dropout(x, p=self.activation_dropout, training=self.training)
        x = self.fc2(x)
        x = F.dropout(x, p=self.dropout, training=self.training)
        x = residual + x
        if not self.normalize_before:
            x = self.final_layer_norm(x)
        return (
            x,
            self_attn_weights,
            layer_state,
        )  # just self_attn weights for now, following t5, layer_state = cache for decoding


class BartDecoder(nn.Module):
    """
    Transformer decoder consisting of *config.decoder_layers* layers. Each layer
    is a :class:`DecoderLayer`.
    Args:
        config: BartConfig



  
  
    
    
  
  


Hope it helps,
Best,